Oct 30 12:44:18.785492 ip-10-0-142-194 systemd[1]: Starting Kubernetes Kubelet... Oct 30 12:44:19.223686 ip-10-0-142-194 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:44:19.223686 ip-10-0-142-194 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 12:44:19.223686 ip-10-0-142-194 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:44:19.223686 ip-10-0-142-194 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:44:19.229115 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229035 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:44:19.229115 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229114 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:44:19.229115 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229119 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:44:19.229212 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229122 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:44:19.229212 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229125 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:44:19.229212 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229128 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:44:19.229212 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229131 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:44:19.229212 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229134 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:44:19.229212 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229137 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:44:19.229212 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229140 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:44:19.229212 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229142 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:44:19.229212 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229145 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:44:19.229212 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229148 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:44:19.229212 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229150 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:44:19.229212 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229153 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:44:19.229212 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229155 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:44:19.229212 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229158 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:44:19.229212 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229161 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:44:19.229212 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229163 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:44:19.229212 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229166 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:44:19.229212 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229172 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:44:19.229212 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229175 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:44:19.229212 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229178 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:44:19.229699 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229180 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:44:19.229699 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229183 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:44:19.229699 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229185 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:44:19.229699 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229188 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:44:19.229699 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229191 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:44:19.229699 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229193 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:44:19.229699 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229196 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:44:19.229699 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229199 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:44:19.229699 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229201 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:44:19.229699 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229204 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:44:19.229699 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229207 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:44:19.229699 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229210 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:44:19.229699 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229213 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:44:19.229699 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229216 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:44:19.229699 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229236 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:44:19.229699 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229239 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:44:19.229699 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229242 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:44:19.229699 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229245 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:44:19.229699 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229250 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:44:19.229699 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229255 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:44:19.230169 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229258 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:44:19.230169 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229261 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:44:19.230169 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229264 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:44:19.230169 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229267 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:44:19.230169 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229270 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:44:19.230169 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229273 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:44:19.230169 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229276 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:44:19.230169 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229279 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:44:19.230169 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229282 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:44:19.230169 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229284 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:44:19.230169 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229287 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:44:19.230169 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229290 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:44:19.230169 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229294 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:44:19.230169 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229297 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:44:19.230169 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229301 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:44:19.230169 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229304 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:44:19.230169 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229307 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:44:19.230169 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229309 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:44:19.230169 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229312 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:44:19.230636 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229314 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:44:19.230636 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229317 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:44:19.230636 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229321 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:44:19.230636 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229323 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:44:19.230636 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229327 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:44:19.230636 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229329 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:44:19.230636 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229333 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:44:19.230636 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229336 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:44:19.230636 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229338 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:44:19.230636 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229341 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:44:19.230636 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229343 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:44:19.230636 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229347 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:44:19.230636 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229349 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:44:19.230636 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229352 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:44:19.230636 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229355 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:44:19.230636 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229358 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:44:19.230636 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229361 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:44:19.230636 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229364 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:44:19.230636 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229366 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:44:19.231096 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229369 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:44:19.231096 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229371 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:44:19.231096 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229374 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:44:19.231096 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229377 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:44:19.231096 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229379 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:44:19.231096 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229382 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:44:19.231096 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229385 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:44:19.231096 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229388 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:44:19.231096 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229391 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:44:19.231096 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229393 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:44:19.231096 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229396 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:44:19.231096 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229399 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:44:19.231096 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229402 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:44:19.231096 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229404 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:44:19.231096 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229407 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:44:19.231096 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229409 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:44:19.231096 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229412 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:44:19.231096 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229415 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:44:19.231096 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229418 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:44:19.231096 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229421 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:44:19.231616 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229423 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:44:19.231616 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229426 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:44:19.231616 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229429 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:44:19.231616 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229431 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:44:19.231616 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.229434 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:44:19.231616 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230245 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:44:19.231616 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230253 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:44:19.231616 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230256 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:44:19.231616 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230260 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:44:19.231616 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230264 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:44:19.231616 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230267 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:44:19.231616 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230269 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:44:19.231616 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230272 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:44:19.231616 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230274 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:44:19.231616 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230277 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:44:19.231616 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230280 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:44:19.231616 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230283 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:44:19.231616 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230285 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:44:19.231616 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230288 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:44:19.232110 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230290 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:44:19.232110 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230293 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:44:19.232110 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230296 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:44:19.232110 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230298 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:44:19.232110 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230301 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:44:19.232110 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230303 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:44:19.232110 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230306 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:44:19.232110 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230308 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:44:19.232110 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230311 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:44:19.232110 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230314 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:44:19.232110 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230319 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:44:19.232110 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230323 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:44:19.232110 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230327 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:44:19.232110 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230331 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:44:19.232110 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230334 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:44:19.232110 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230337 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:44:19.232110 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230339 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:44:19.232110 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230342 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:44:19.232110 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230345 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:44:19.232110 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230348 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:44:19.232617 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230350 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:44:19.232617 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230353 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:44:19.232617 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230356 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:44:19.232617 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230359 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:44:19.232617 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230362 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:44:19.232617 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230364 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:44:19.232617 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230368 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:44:19.232617 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230370 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:44:19.232617 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230373 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:44:19.232617 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230376 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:44:19.232617 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230378 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:44:19.232617 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230381 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:44:19.232617 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230383 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:44:19.232617 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230386 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:44:19.232617 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230389 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:44:19.232617 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230391 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:44:19.232617 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230394 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:44:19.232617 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230397 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:44:19.232617 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230400 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:44:19.232617 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230403 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:44:19.233181 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230405 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:44:19.233181 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230408 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:44:19.233181 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230411 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:44:19.233181 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230413 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:44:19.233181 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230416 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:44:19.233181 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230419 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:44:19.233181 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230422 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:44:19.233181 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230424 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:44:19.233181 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230427 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:44:19.233181 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230429 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:44:19.233181 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230432 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:44:19.233181 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230435 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:44:19.233181 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230437 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:44:19.233181 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230440 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:44:19.233181 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230443 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:44:19.233181 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230446 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:44:19.233181 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230449 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:44:19.233181 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230452 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:44:19.233181 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230454 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:44:19.233664 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230457 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:44:19.233664 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230459 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:44:19.233664 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230462 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:44:19.233664 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230465 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:44:19.233664 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230467 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:44:19.233664 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230470 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:44:19.233664 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230472 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:44:19.233664 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230475 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:44:19.233664 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230477 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:44:19.233664 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230480 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:44:19.233664 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230483 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:44:19.233664 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230486 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:44:19.233664 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230488 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:44:19.233664 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230492 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:44:19.233664 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230496 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:44:19.233664 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230500 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:44:19.233664 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230502 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:44:19.233664 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230505 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:44:19.233664 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230508 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:44:19.234113 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230510 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:44:19.234113 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230513 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:44:19.234113 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230515 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:44:19.234113 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230518 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:44:19.234113 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230520 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:44:19.234113 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230523 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:44:19.234113 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230525 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:44:19.234113 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230528 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:44:19.234113 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230530 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:44:19.234113 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230533 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:44:19.234113 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230536 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:44:19.234113 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230539 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:44:19.234113 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230542 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:44:19.234113 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.230544 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:44:19.234113 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231042 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:44:19.234113 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231048 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:44:19.234113 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231050 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:44:19.234113 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231054 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:44:19.234113 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231057 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:44:19.234576 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231059 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:44:19.234576 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231062 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:44:19.234576 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231065 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:44:19.234576 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231068 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:44:19.234576 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231071 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:44:19.234576 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231074 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:44:19.234576 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231077 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:44:19.234576 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231079 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:44:19.234576 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231082 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:44:19.234576 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231085 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:44:19.234576 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231088 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:44:19.234576 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231090 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:44:19.234576 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231093 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:44:19.234576 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231095 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:44:19.234576 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231098 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:44:19.234576 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231101 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:44:19.234576 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231104 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:44:19.234576 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231108 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:44:19.234576 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231112 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:44:19.235048 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231115 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:44:19.235048 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231118 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:44:19.235048 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231120 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:44:19.235048 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231123 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:44:19.235048 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231126 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:44:19.235048 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231129 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:44:19.235048 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231132 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:44:19.235048 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231134 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:44:19.235048 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231137 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:44:19.235048 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231139 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:44:19.235048 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231142 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:44:19.235048 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231145 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:44:19.235048 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231147 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:44:19.235048 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231150 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:44:19.235048 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231152 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:44:19.235048 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231155 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:44:19.235048 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231159 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:44:19.235048 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231162 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:44:19.235048 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231164 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:44:19.235048 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231167 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:44:19.235544 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231169 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:44:19.235544 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231172 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:44:19.235544 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231174 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:44:19.235544 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231177 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:44:19.235544 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231179 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:44:19.235544 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231182 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:44:19.235544 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231185 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:44:19.235544 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231188 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:44:19.235544 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231191 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:44:19.235544 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231193 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:44:19.235544 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231196 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:44:19.235544 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231198 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:44:19.235544 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231201 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:44:19.235544 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231203 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:44:19.235544 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231206 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:44:19.235544 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231208 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:44:19.235544 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231211 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:44:19.235544 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231214 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:44:19.235544 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231229 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:44:19.235544 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231232 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:44:19.235544 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231235 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:44:19.236032 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231237 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:44:19.236032 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231240 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:44:19.236032 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231243 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:44:19.236032 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231246 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:44:19.236032 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231250 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:44:19.236032 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231252 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:44:19.236032 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231256 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:44:19.236032 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231259 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:44:19.236032 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231262 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:44:19.236032 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231265 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:44:19.236032 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231268 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:44:19.236032 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231270 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:44:19.236032 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231273 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:44:19.236032 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231276 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:44:19.236032 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231278 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:44:19.236032 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231281 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:44:19.236032 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231284 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:44:19.236032 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231286 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:44:19.236032 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231289 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:44:19.236032 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231291 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:44:19.236559 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231294 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:44:19.236559 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231296 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:44:19.236559 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231299 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:44:19.236559 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231302 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:44:19.236559 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231304 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:44:19.236559 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231309 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:44:19.236559 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231313 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:44:19.236559 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231316 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:44:19.236559 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231319 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:44:19.236559 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231323 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:44:19.236559 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231326 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:44:19.236559 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231328 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:44:19.236559 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231331 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:44:19.236559 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231333 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:44:19.236559 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231336 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:44:19.236559 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231340 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:44:19.236559 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231343 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:44:19.236559 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231348 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:44:19.236559 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231352 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:44:19.237722 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231355 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:44:19.237722 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.231357 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:44:19.237722 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232470 2574 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 12:44:19.237722 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232480 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 12:44:19.237722 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232487 2574 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 12:44:19.237722 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232492 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 12:44:19.237722 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232497 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 12:44:19.237722 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232500 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 12:44:19.237722 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232507 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 12:44:19.237722 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232511 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 12:44:19.237722 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232515 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 12:44:19.237722 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232518 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 12:44:19.237722 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232522 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 12:44:19.237722 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232525 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 12:44:19.237722 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232528 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 12:44:19.237722 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232531 2574 flags.go:64] FLAG: --cgroup-root="" Oct 30 12:44:19.237722 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232534 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 12:44:19.237722 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232537 2574 flags.go:64] FLAG: --client-ca-file="" Oct 30 12:44:19.237722 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232540 2574 flags.go:64] FLAG: --cloud-provider="external" Oct 30 12:44:19.237722 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232543 2574 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 12:44:19.237722 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232548 2574 flags.go:64] FLAG: --cluster-domain="" Oct 30 12:44:19.237722 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232552 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 12:44:19.237722 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232555 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 12:44:19.238275 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232559 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 12:44:19.238275 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232563 2574 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 12:44:19.238275 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232567 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 12:44:19.238275 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232570 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 12:44:19.238275 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232574 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 12:44:19.238275 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232578 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 12:44:19.238275 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232581 2574 flags.go:64] FLAG: --contention-profiling="false" Oct 30 12:44:19.238275 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232586 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 12:44:19.238275 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232590 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 12:44:19.238275 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232594 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 12:44:19.238275 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232597 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 12:44:19.238275 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232601 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 12:44:19.238275 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232605 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 12:44:19.238275 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232608 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 12:44:19.238275 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232611 2574 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 12:44:19.238275 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232614 2574 flags.go:64] FLAG: --enable-server="true" Oct 30 12:44:19.238275 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232618 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 12:44:19.238275 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232622 2574 flags.go:64] FLAG: --event-burst="100" Oct 30 12:44:19.238275 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232625 2574 flags.go:64] FLAG: --event-qps="50" Oct 30 12:44:19.238275 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232629 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 12:44:19.238275 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232632 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 12:44:19.238275 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232635 2574 flags.go:64] FLAG: --eviction-hard="" Oct 30 12:44:19.238275 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232640 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 12:44:19.238275 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232643 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 12:44:19.238873 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232646 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 12:44:19.238873 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232650 2574 flags.go:64] FLAG: --eviction-soft="" Oct 30 12:44:19.238873 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232653 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 12:44:19.238873 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232656 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 12:44:19.238873 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232659 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 12:44:19.238873 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232662 2574 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 12:44:19.238873 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232665 2574 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 12:44:19.238873 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232668 2574 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 12:44:19.238873 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232671 2574 flags.go:64] FLAG: --feature-gates="" Oct 30 12:44:19.238873 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232676 2574 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 12:44:19.238873 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232680 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 12:44:19.238873 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232683 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 12:44:19.238873 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232687 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 12:44:19.238873 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232690 2574 flags.go:64] FLAG: --healthz-port="10248" Oct 30 12:44:19.238873 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232693 2574 flags.go:64] FLAG: --help="false" Oct 30 12:44:19.238873 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232698 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-142-194.ec2.internal" Oct 30 12:44:19.238873 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232702 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 12:44:19.238873 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232705 2574 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 12:44:19.238873 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232708 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 12:44:19.238873 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232712 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 12:44:19.238873 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232716 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 12:44:19.238873 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232719 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 12:44:19.238873 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232722 2574 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 12:44:19.239458 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232725 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 12:44:19.239458 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232728 2574 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 12:44:19.239458 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232731 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 12:44:19.239458 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232734 2574 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 12:44:19.239458 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232737 2574 flags.go:64] FLAG: --kube-reserved="" Oct 30 12:44:19.239458 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232740 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 12:44:19.239458 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232743 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 12:44:19.239458 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232747 2574 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 12:44:19.239458 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232749 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 12:44:19.239458 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232752 2574 flags.go:64] FLAG: --lock-file="" Oct 30 12:44:19.239458 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232755 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 12:44:19.239458 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232758 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 12:44:19.239458 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232761 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 12:44:19.239458 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232767 2574 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 12:44:19.239458 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232770 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 12:44:19.239458 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232773 2574 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 12:44:19.239458 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232776 2574 flags.go:64] FLAG: --logging-format="text" Oct 30 12:44:19.239458 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232779 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 12:44:19.239458 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232782 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 12:44:19.239458 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232786 2574 flags.go:64] FLAG: --manifest-url="" Oct 30 12:44:19.239458 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232789 2574 flags.go:64] FLAG: --manifest-url-header="" Oct 30 12:44:19.239458 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232794 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 12:44:19.239458 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232797 2574 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 12:44:19.239458 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232801 2574 flags.go:64] FLAG: --max-pods="110" Oct 30 12:44:19.239458 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232806 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 12:44:19.240066 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232809 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 12:44:19.240066 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232812 2574 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 12:44:19.240066 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232815 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 12:44:19.240066 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232818 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 12:44:19.240066 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232821 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 12:44:19.240066 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232824 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 12:44:19.240066 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232831 2574 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 12:44:19.240066 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232835 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 12:44:19.240066 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232838 2574 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 12:44:19.240066 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232841 2574 flags.go:64] FLAG: --pod-cidr="" Oct 30 12:44:19.240066 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232844 2574 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 12:44:19.240066 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232847 2574 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 12:44:19.240066 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232850 2574 flags.go:64] FLAG: --pods-per-core="0" Oct 30 12:44:19.240066 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232853 2574 flags.go:64] FLAG: --port="10250" Oct 30 12:44:19.240066 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232856 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 12:44:19.240066 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232858 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-035fb9a9971d8c7ef" Oct 30 12:44:19.240066 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232862 2574 flags.go:64] FLAG: --qos-reserved="" Oct 30 12:44:19.240066 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232864 2574 flags.go:64] FLAG: --read-only-port="10255" Oct 30 12:44:19.240066 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232867 2574 flags.go:64] FLAG: --register-node="true" Oct 30 12:44:19.240066 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232870 2574 flags.go:64] FLAG: --register-with-taints="" Oct 30 12:44:19.240066 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232874 2574 flags.go:64] FLAG: --registry-burst="10" Oct 30 12:44:19.240066 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232877 2574 flags.go:64] FLAG: --registry-qps="5" Oct 30 12:44:19.240066 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232880 2574 flags.go:64] FLAG: --reserved-cpus="" Oct 30 12:44:19.240066 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232883 2574 flags.go:64] FLAG: --reserved-memory="" Oct 30 12:44:19.240066 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232886 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 12:44:19.240066 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232890 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 12:44:19.240705 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232893 2574 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 12:44:19.240705 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232896 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 12:44:19.240705 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232899 2574 flags.go:64] FLAG: --runonce="false" Oct 30 12:44:19.240705 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232902 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 12:44:19.240705 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232906 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 12:44:19.240705 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232910 2574 flags.go:64] FLAG: --seccomp-default="false" Oct 30 12:44:19.240705 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232913 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 12:44:19.240705 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232916 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 12:44:19.240705 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232920 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 12:44:19.240705 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232923 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 12:44:19.240705 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232926 2574 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 12:44:19.240705 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232929 2574 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 12:44:19.240705 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232932 2574 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 12:44:19.240705 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232935 2574 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 12:44:19.240705 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232938 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 12:44:19.240705 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232941 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 12:44:19.240705 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232944 2574 flags.go:64] FLAG: --system-cgroups="" Oct 30 12:44:19.240705 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232947 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 12:44:19.240705 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232953 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 12:44:19.240705 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232956 2574 flags.go:64] FLAG: --tls-cert-file="" Oct 30 12:44:19.240705 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232959 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 12:44:19.240705 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232964 2574 flags.go:64] FLAG: --tls-min-version="" Oct 30 12:44:19.240705 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232966 2574 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 12:44:19.240705 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232969 2574 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 12:44:19.240705 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232972 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 12:44:19.241361 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232975 2574 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 12:44:19.241361 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232979 2574 flags.go:64] FLAG: --v="2" Oct 30 12:44:19.241361 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232983 2574 flags.go:64] FLAG: --version="false" Oct 30 12:44:19.241361 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232987 2574 flags.go:64] FLAG: --vmodule="" Oct 30 12:44:19.241361 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232991 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 12:44:19.241361 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.232995 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 12:44:19.241361 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233135 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:44:19.241361 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233139 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:44:19.241361 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233144 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:44:19.241361 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233149 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:44:19.241361 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233152 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:44:19.241361 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233156 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:44:19.241361 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233161 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:44:19.241361 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233164 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:44:19.241361 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233168 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:44:19.241361 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233171 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:44:19.241361 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233173 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:44:19.241361 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233176 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:44:19.241361 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233179 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:44:19.241361 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233182 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:44:19.241843 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233185 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:44:19.241843 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233187 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:44:19.241843 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233190 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:44:19.241843 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233194 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:44:19.241843 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233196 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:44:19.241843 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233199 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:44:19.241843 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233202 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:44:19.241843 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233205 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:44:19.241843 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233207 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:44:19.241843 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233210 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:44:19.241843 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233213 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:44:19.241843 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233215 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:44:19.241843 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233229 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:44:19.241843 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233232 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:44:19.241843 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233235 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:44:19.241843 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233238 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:44:19.241843 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233241 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:44:19.241843 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233243 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:44:19.241843 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233246 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:44:19.242298 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233248 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:44:19.242298 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233254 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:44:19.242298 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233257 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:44:19.242298 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233260 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:44:19.242298 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233263 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:44:19.242298 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233266 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:44:19.242298 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233269 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:44:19.242298 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233272 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:44:19.242298 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233274 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:44:19.242298 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233277 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:44:19.242298 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233280 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:44:19.242298 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233282 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:44:19.242298 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233285 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:44:19.242298 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233288 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:44:19.242298 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233291 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:44:19.242298 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233293 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:44:19.242298 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233296 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:44:19.242298 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233299 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:44:19.242298 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233301 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:44:19.242298 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233304 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:44:19.242773 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233307 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:44:19.242773 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233309 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:44:19.242773 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233312 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:44:19.242773 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233315 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:44:19.242773 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233317 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:44:19.242773 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233320 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:44:19.242773 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233323 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:44:19.242773 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233325 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:44:19.242773 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233328 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:44:19.242773 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233331 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:44:19.242773 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233333 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:44:19.242773 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233336 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:44:19.242773 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233338 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:44:19.242773 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233345 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:44:19.242773 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233347 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:44:19.242773 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233350 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:44:19.242773 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233353 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:44:19.242773 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233356 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:44:19.242773 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233359 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:44:19.242773 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233361 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:44:19.243275 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233364 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:44:19.243275 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233366 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:44:19.243275 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233369 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:44:19.243275 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233372 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:44:19.243275 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233375 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:44:19.243275 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233377 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:44:19.243275 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233380 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:44:19.243275 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233382 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:44:19.243275 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233385 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:44:19.243275 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233387 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:44:19.243275 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233390 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:44:19.243275 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233392 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:44:19.243275 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233395 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:44:19.243275 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233397 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:44:19.243275 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233401 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:44:19.243275 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233403 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:44:19.243275 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233406 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:44:19.243275 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233408 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:44:19.243275 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233411 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:44:19.243724 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233414 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:44:19.243724 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233416 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:44:19.243724 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233419 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:44:19.243724 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233421 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:44:19.243724 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233424 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:44:19.243724 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233426 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:44:19.243724 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233430 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:44:19.243724 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233434 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:44:19.243724 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233438 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:44:19.243724 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233441 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:44:19.243724 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233445 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:44:19.243724 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233447 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:44:19.243724 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233450 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:44:19.243724 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233453 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:44:19.243724 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.233479 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:44:19.243724 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233519 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:44:19.244102 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233523 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:44:19.244102 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233526 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:44:19.244102 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233529 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:44:19.244102 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233533 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:44:19.244102 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233536 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:44:19.244102 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233539 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:44:19.244102 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233542 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:44:19.244102 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233544 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:44:19.244102 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233547 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:44:19.244102 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233549 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:44:19.244102 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233552 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:44:19.244102 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233555 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:44:19.244102 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233557 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:44:19.244102 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233560 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:44:19.244102 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233562 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:44:19.244102 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233565 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:44:19.244102 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233568 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:44:19.244102 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233570 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:44:19.244102 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233573 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:44:19.244102 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233576 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:44:19.244634 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233578 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:44:19.244634 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233581 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:44:19.244634 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233583 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:44:19.244634 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233587 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:44:19.244634 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233590 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:44:19.244634 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233593 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:44:19.244634 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233595 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:44:19.244634 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233599 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:44:19.244634 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233602 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:44:19.244634 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233604 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:44:19.244634 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233607 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:44:19.244634 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233609 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:44:19.244634 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233612 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:44:19.244634 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233615 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:44:19.244634 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233617 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:44:19.244634 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233620 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:44:19.244634 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233623 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:44:19.244634 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233626 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:44:19.244634 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233628 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:44:19.244634 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233631 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:44:19.245112 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233634 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:44:19.245112 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233636 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:44:19.245112 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233639 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:44:19.245112 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233641 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:44:19.245112 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233644 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:44:19.245112 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233646 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:44:19.245112 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233649 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:44:19.245112 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233652 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:44:19.245112 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233654 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:44:19.245112 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233657 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:44:19.245112 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233659 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:44:19.245112 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233662 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:44:19.245112 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233665 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:44:19.245112 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233667 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:44:19.245112 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233670 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:44:19.245112 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233674 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:44:19.245112 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233677 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:44:19.245112 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233680 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:44:19.245112 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233683 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:44:19.245601 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233686 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:44:19.245601 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233689 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:44:19.245601 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233692 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:44:19.245601 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233694 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:44:19.245601 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233697 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:44:19.245601 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233700 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:44:19.245601 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233702 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:44:19.245601 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233705 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:44:19.245601 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233708 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:44:19.245601 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233711 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:44:19.245601 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233715 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:44:19.245601 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233718 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:44:19.245601 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233721 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:44:19.245601 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233724 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:44:19.245601 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233726 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:44:19.245601 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233729 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:44:19.245601 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233731 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:44:19.245601 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233734 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:44:19.245601 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233738 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:44:19.246059 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233742 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:44:19.246059 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233745 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:44:19.246059 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233748 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:44:19.246059 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233751 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:44:19.246059 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233753 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:44:19.246059 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233756 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:44:19.246059 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233759 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:44:19.246059 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233761 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:44:19.246059 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233764 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:44:19.246059 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233767 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:44:19.246059 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233770 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:44:19.246059 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233773 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:44:19.246059 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233777 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:44:19.246059 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233780 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:44:19.246059 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233782 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:44:19.246059 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233785 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:44:19.246059 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233788 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:44:19.246059 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233790 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:44:19.246059 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233793 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:44:19.246557 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233795 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:44:19.246557 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233798 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:44:19.246557 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233801 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:44:19.246557 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233803 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:44:19.246557 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233806 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:44:19.246557 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233810 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:44:19.246557 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233812 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:44:19.246557 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.233815 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:44:19.246557 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.241022 2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 12:44:19.246557 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.241039 2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 12:44:19.246557 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.241745 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 12:44:19.246557 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.241752 2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 12:44:19.246557 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241813 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:44:19.246557 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241818 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:44:19.246557 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241821 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:44:19.246557 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241824 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:44:19.246557 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241828 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:44:19.246557 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241830 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:44:19.246557 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241833 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:44:19.247035 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241836 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:44:19.247035 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241840 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:44:19.247035 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241844 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:44:19.247035 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241847 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:44:19.247035 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241850 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:44:19.247035 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241854 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:44:19.247035 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241856 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:44:19.247035 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241859 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:44:19.247035 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241862 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:44:19.247035 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241865 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:44:19.247035 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241867 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:44:19.247035 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241870 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:44:19.247035 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241873 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:44:19.247035 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241876 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:44:19.247035 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241878 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:44:19.247035 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241881 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:44:19.247035 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241884 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:44:19.247035 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241886 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:44:19.247035 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241889 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:44:19.247496 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241892 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:44:19.247496 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241894 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:44:19.247496 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241897 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:44:19.247496 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241902 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:44:19.247496 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241905 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:44:19.247496 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241908 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:44:19.247496 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241911 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:44:19.247496 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241914 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:44:19.247496 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241916 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:44:19.247496 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241919 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:44:19.247496 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241922 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:44:19.247496 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241924 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:44:19.247496 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241927 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:44:19.247496 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241929 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:44:19.247496 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241932 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:44:19.247496 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241935 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:44:19.247496 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241937 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:44:19.247496 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241940 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:44:19.247496 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241944 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:44:19.247496 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241946 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:44:19.247964 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241949 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:44:19.247964 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241952 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:44:19.247964 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241955 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:44:19.247964 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241957 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:44:19.247964 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241960 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:44:19.247964 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241962 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:44:19.247964 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241965 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:44:19.247964 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241968 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:44:19.247964 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241970 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:44:19.247964 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241973 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:44:19.247964 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241975 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:44:19.247964 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241978 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:44:19.247964 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241981 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:44:19.247964 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241983 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:44:19.247964 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241986 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:44:19.247964 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241989 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:44:19.247964 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241992 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:44:19.247964 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.241996 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:44:19.247964 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242000 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:44:19.248517 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242003 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:44:19.248517 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242006 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:44:19.248517 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242009 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:44:19.248517 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242011 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:44:19.248517 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242014 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:44:19.248517 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242017 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:44:19.248517 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242020 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:44:19.248517 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242022 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:44:19.248517 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242025 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:44:19.248517 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242027 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:44:19.248517 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242030 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:44:19.248517 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242032 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:44:19.248517 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242035 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:44:19.248517 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242038 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:44:19.248517 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242040 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:44:19.248517 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242043 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:44:19.248517 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242045 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:44:19.248517 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242048 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:44:19.248517 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242050 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:44:19.248517 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242053 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:44:19.249000 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242055 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:44:19.249000 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242058 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:44:19.249000 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242060 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:44:19.249000 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242063 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:44:19.249000 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242066 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:44:19.249000 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242068 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:44:19.249000 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242071 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:44:19.249000 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242074 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:44:19.249000 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242077 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:44:19.249000 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242080 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:44:19.249000 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242083 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:44:19.249000 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242085 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:44:19.249000 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242088 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:44:19.249000 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242091 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:44:19.249000 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242093 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:44:19.249000 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242096 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:44:19.249000 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242099 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:44:19.249000 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242101 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:44:19.249000 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242104 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:44:19.249000 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242106 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:44:19.249485 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242109 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:44:19.249485 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.242136 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:44:19.249485 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242287 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:44:19.249485 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242292 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:44:19.249485 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242295 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:44:19.249485 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242298 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:44:19.249485 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242301 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:44:19.249485 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242304 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:44:19.249485 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242306 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:44:19.249485 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242309 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:44:19.249485 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242312 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:44:19.249485 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242314 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:44:19.249485 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242317 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:44:19.249485 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242320 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:44:19.249485 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242324 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:44:19.249934 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242328 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:44:19.249934 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242332 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:44:19.249934 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242335 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:44:19.249934 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242338 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:44:19.249934 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242341 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:44:19.249934 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242344 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:44:19.249934 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242347 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:44:19.249934 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242350 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:44:19.249934 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242353 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:44:19.249934 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242356 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:44:19.249934 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242359 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:44:19.249934 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242362 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:44:19.249934 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242365 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:44:19.249934 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242369 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:44:19.249934 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242372 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:44:19.249934 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242376 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:44:19.249934 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242378 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:44:19.249934 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242381 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:44:19.249934 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242383 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:44:19.250473 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242386 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:44:19.250473 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242389 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:44:19.250473 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242391 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:44:19.250473 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242394 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:44:19.250473 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242396 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:44:19.250473 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242399 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:44:19.250473 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242401 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:44:19.250473 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242404 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:44:19.250473 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242406 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:44:19.250473 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242409 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:44:19.250473 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242412 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:44:19.250473 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242414 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:44:19.250473 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242417 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:44:19.250473 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242419 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:44:19.250473 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242422 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:44:19.250473 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242425 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:44:19.250473 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242427 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:44:19.250473 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242430 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:44:19.250473 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242433 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:44:19.250473 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242435 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:44:19.251130 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242438 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:44:19.251130 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242441 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:44:19.251130 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242443 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:44:19.251130 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242446 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:44:19.251130 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242448 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:44:19.251130 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242452 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:44:19.251130 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242454 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:44:19.251130 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242457 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:44:19.251130 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242459 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:44:19.251130 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242462 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:44:19.251130 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242464 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:44:19.251130 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242467 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:44:19.251130 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242470 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:44:19.251130 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242472 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:44:19.251130 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242474 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:44:19.251130 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242477 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:44:19.251130 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242480 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:44:19.251130 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242482 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:44:19.251130 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242485 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:44:19.251130 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242487 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:44:19.251667 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242490 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:44:19.251667 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242492 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:44:19.251667 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242495 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:44:19.251667 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242498 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:44:19.251667 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242504 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:44:19.251667 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242507 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:44:19.251667 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242509 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:44:19.251667 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242512 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:44:19.251667 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242515 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:44:19.251667 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242517 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:44:19.251667 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242520 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:44:19.251667 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242523 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:44:19.251667 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242525 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:44:19.251667 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242527 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:44:19.251667 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242530 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:44:19.251667 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242533 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:44:19.251667 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242536 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:44:19.251667 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242538 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:44:19.251667 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242541 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:44:19.252114 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242543 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:44:19.252114 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242546 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:44:19.252114 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242548 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:44:19.252114 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242551 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:44:19.252114 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242553 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:44:19.252114 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242556 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:44:19.252114 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242559 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:44:19.252114 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242561 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:44:19.252114 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242564 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:44:19.252114 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242566 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:44:19.252114 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242569 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:44:19.252114 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242571 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:44:19.252114 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242574 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:44:19.252114 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242577 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:44:19.252114 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242579 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:44:19.252114 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.242603 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:44:19.252589 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242634 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:44:19.252589 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242638 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:44:19.252589 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242643 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:44:19.252589 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242646 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:44:19.252589 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242648 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:44:19.252589 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242651 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:44:19.252589 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242654 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:44:19.252589 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242656 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:44:19.252589 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242659 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:44:19.252589 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242662 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:44:19.252589 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242664 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:44:19.252589 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242667 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:44:19.252589 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242670 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:44:19.252589 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242673 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:44:19.252589 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242675 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:44:19.252589 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242678 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:44:19.252589 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242681 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:44:19.252589 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242684 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:44:19.252589 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242686 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:44:19.252589 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242689 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:44:19.253043 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242691 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:44:19.253043 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242694 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:44:19.253043 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242697 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:44:19.253043 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242699 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:44:19.253043 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242702 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:44:19.253043 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242704 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:44:19.253043 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242707 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:44:19.253043 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242709 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:44:19.253043 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242712 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:44:19.253043 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242715 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:44:19.253043 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242717 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:44:19.253043 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242720 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:44:19.253043 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242723 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:44:19.253043 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242726 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:44:19.253043 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242730 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:44:19.253043 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242732 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:44:19.253043 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242735 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:44:19.253043 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242737 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:44:19.253043 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242740 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:44:19.253523 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242743 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:44:19.253523 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242745 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:44:19.253523 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242748 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:44:19.253523 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242750 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:44:19.253523 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242753 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:44:19.253523 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242755 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:44:19.253523 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242758 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:44:19.253523 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242761 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:44:19.253523 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242764 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:44:19.253523 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242767 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:44:19.253523 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242770 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:44:19.253523 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242772 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:44:19.253523 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242775 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:44:19.253523 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242778 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:44:19.253523 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242780 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:44:19.253523 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242784 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:44:19.253523 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242786 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:44:19.253523 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242789 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:44:19.253523 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242792 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:44:19.253523 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242794 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:44:19.253984 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242797 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:44:19.253984 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242799 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:44:19.253984 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242802 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:44:19.253984 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242805 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:44:19.253984 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242808 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:44:19.253984 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242810 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:44:19.253984 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242813 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:44:19.253984 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242816 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:44:19.253984 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242829 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:44:19.253984 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242833 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:44:19.253984 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242837 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:44:19.253984 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242841 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:44:19.253984 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242844 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:44:19.253984 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242847 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:44:19.253984 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242849 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:44:19.253984 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242852 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:44:19.253984 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242855 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:44:19.253984 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242857 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:44:19.253984 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242860 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:44:19.254439 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242862 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:44:19.254439 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242866 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:44:19.254439 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242870 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:44:19.254439 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242874 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:44:19.254439 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242877 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:44:19.254439 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242880 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:44:19.254439 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242882 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:44:19.254439 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242885 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:44:19.254439 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242888 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:44:19.254439 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242890 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:44:19.254439 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242893 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:44:19.254439 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242895 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:44:19.254439 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242898 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:44:19.254439 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242900 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:44:19.254439 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242903 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:44:19.254439 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242906 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:44:19.254439 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242908 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:44:19.254439 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242911 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:44:19.254439 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242914 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:44:19.254953 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242917 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:44:19.254953 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242920 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:44:19.254953 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242923 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:44:19.254953 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242926 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:44:19.254953 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242928 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:44:19.254953 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242931 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:44:19.254953 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242933 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:44:19.254953 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242936 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:44:19.254953 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:19.242938 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:44:19.254953 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.242995 2574 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 12:44:19.254953 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.245429 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 12:44:19.254953 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.246540 2574 server.go:1014] "Starting client certificate rotation" Oct 30 12:44:19.254953 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.246634 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 12:44:19.254953 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.247478 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 12:44:19.254953 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.248700 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 12:44:19.254953 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.253200 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 12:44:19.263404 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.263387 2574 log.go:25] "Validated CRI v1 runtime API" Oct 30 12:44:19.267929 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.267908 2574 log.go:25] "Validated CRI v1 image API" Oct 30 12:44:19.268724 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.268701 2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 12:44:19.271582 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.271561 2574 fs.go:135] Filesystem UUIDs: map[1de3ac00-83a9-42cb-83d5-8961964fc912:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b2c69896-e7a2-41a5-abde-6b98e042ad92:/dev/nvme0n1p4] Oct 30 12:44:19.271648 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.271581 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 12:44:19.277061 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.276021 2574 manager.go:217] Machine: {Timestamp:2026-10-30 12:44:19.27555869 +0000 UTC m=+0.375879495 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3126436 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec228b8fd08c0c68e4e5dde1421ab5ea SystemUUID:ec228b8f-d08c-0c68-e4e5-dde1421ab5ea BootID:d46a3f6c-4b13-4bbb-a557-471599694e2e Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d2:13:67:4d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d2:13:67:4d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:42:77:51:7b:0c:12 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 12:44:19.277744 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.277733 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 12:44:19.277857 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.277845 2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 12:44:19.281463 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.281438 2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 12:44:19.281589 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.281465 2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-194.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 12:44:19.281630 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.281591 2574 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 12:44:19.281630 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.281600 2574 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 12:44:19.281630 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.281622 2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 12:44:19.282671 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.282662 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 12:44:19.283958 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.283948 2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 12:44:19.288572 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.288561 2574 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 12:44:19.288688 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.288679 2574 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 12:44:19.289489 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.289479 2574 kubelet.go:493] "Attempting to sync node with API server" Oct 30 12:44:19.289520 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.289495 2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 12:44:19.289520 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.289506 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 12:44:19.289520 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.289515 2574 kubelet.go:405] "Adding apiserver pod source" Oct 30 12:44:19.289619 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.289524 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 12:44:19.289876 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.289864 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 12:44:19.292594 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.292562 2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 12:44:19.299782 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.299768 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 12:44:19.300641 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.300629 2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 12:44:19.300680 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.300660 2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 12:44:19.300827 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.300816 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 12:44:19.300862 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.300834 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 12:44:19.300862 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.300841 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 12:44:19.300862 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.300847 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 12:44:19.300862 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.300853 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 12:44:19.300862 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.300860 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 12:44:19.301620 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.301611 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 12:44:19.301653 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.301620 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 12:44:19.301653 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.301628 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 12:44:19.301653 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.301633 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 12:44:19.301653 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.301643 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 12:44:19.301653 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.301651 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 12:44:19.302517 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.302507 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 12:44:19.302550 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.302520 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 12:44:19.308631 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.308610 2574 server.go:1263] "Started kubelet" Oct 30 12:44:19.308744 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.308711 2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 12:44:19.308879 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.308830 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 12:44:19.308935 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.308901 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 12:44:19.309466 ip-10-0-142-194 systemd[1]: Started Kubernetes Kubelet. Oct 30 12:44:19.310544 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.310528 2574 server.go:317] "Adding debug handlers to kubelet server" Oct 30 12:44:19.310629 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.310574 2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 12:44:19.314723 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:19.314706 2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 12:44:19.318551 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.318533 2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 12:44:19.318629 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.318551 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 12:44:19.318629 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:19.318601 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-194.ec2.internal\" not found" Oct 30 12:44:19.319860 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.319828 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 12:44:19.319860 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.319829 2574 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 12:44:19.319860 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.319858 2574 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 12:44:19.320053 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.319977 2574 reconstruct.go:97] "Volume reconstruction finished" Oct 30 12:44:19.320053 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.319985 2574 reconciler.go:29] "Reconciler: start to sync state" Oct 30 12:44:19.320154 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.320138 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 12:44:19.320154 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.320154 2574 factory.go:55] Registering systemd factory Oct 30 12:44:19.320263 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.320160 2574 factory.go:223] Registration of the systemd container factory successfully Oct 30 12:44:19.320380 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.320367 2574 factory.go:153] Registering CRI-O factory Oct 30 12:44:19.320445 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.320384 2574 factory.go:223] Registration of the crio container factory successfully Oct 30 12:44:19.320445 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.320409 2574 factory.go:103] Registering Raw factory Oct 30 12:44:19.320445 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.320426 2574 manager.go:1199] Started watching for new ooms in manager Oct 30 12:44:19.321495 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.321481 2574 manager.go:319] Starting recovery of all containers Oct 30 12:44:19.331315 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.331297 2574 manager.go:324] Recovery completed Oct 30 12:44:19.339710 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.339693 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:44:19.339935 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.339916 2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 12:44:19.341506 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.341486 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:44:19.341590 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.341522 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:44:19.341590 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.341536 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:44:19.342116 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.342101 2574 cpu_manager.go:226] "Starting" policy="none" Oct 30 12:44:19.342116 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.342110 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:44:19.342257 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.342114 2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 12:44:19.342257 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.342179 2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 12:44:19.342912 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.342893 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:44:19.342979 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.342927 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:44:19.342979 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.342942 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:44:19.344573 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.344556 2574 policy_none.go:50] "Start" Oct 30 12:44:19.344660 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.344579 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 12:44:19.344660 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.344594 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 12:44:19.344753 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.344558 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-194.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 12:44:19.345717 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:19.344552 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-194.ec2.internal.18e34f4accf7da6e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-194.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-194.ec2.internal,},FirstTimestamp:2026-10-30 12:44:19.308558958 +0000 UTC m=+0.408879763,LastTimestamp:2026-10-30 12:44:19.308558958 +0000 UTC m=+0.408879763,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-194.ec2.internal,}" Oct 30 12:44:19.346658 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.346645 2574 policy_none.go:44] "Start" Oct 30 12:44:19.366552 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:19.366439 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-194.ec2.internal.18e34f4aceee8f3f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-194.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-194.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-194.ec2.internal,},FirstTimestamp:2026-10-30 12:44:19.341504319 +0000 UTC m=+0.441825125,LastTimestamp:2026-10-30 12:44:19.341504319 +0000 UTC m=+0.441825125,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-194.ec2.internal,}" Oct 30 12:44:19.366552 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:19.366538 2574 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-194.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 12:44:19.369426 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:19.367338 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 12:44:19.369426 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:19.367338 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-194.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 12:44:19.400079 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.377754 2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 12:44:19.400079 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:19.379784 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 12:44:19.400079 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.380389 2574 manager.go:341] "Starting Device Plugin manager" Oct 30 12:44:19.400079 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:19.380414 2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 12:44:19.400079 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.380422 2574 server.go:87] "Starting device plugin registration server" Oct 30 12:44:19.400079 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.380613 2574 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 12:44:19.400079 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.380621 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 12:44:19.400079 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.380721 2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 12:44:19.400079 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.380802 2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 12:44:19.400079 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.380810 2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 12:44:19.400079 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:19.381423 2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 12:44:19.400079 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:19.381472 2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-194.ec2.internal\" not found" Oct 30 12:44:19.400079 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.386072 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 12:44:19.400079 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.387286 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 12:44:19.400079 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.387304 2574 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 12:44:19.400079 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.387325 2574 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 12:44:19.400079 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:19.387365 2574 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 12:44:19.400079 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:19.387335 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-194.ec2.internal.18e34f4aceeeed2a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-194.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-194.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-194.ec2.internal,},FirstTimestamp:2026-10-30 12:44:19.341528362 +0000 UTC m=+0.441849169,LastTimestamp:2026-10-30 12:44:19.341528362 +0000 UTC m=+0.441849169,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-194.ec2.internal,}" Oct 30 12:44:19.400079 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.393157 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wkzj5" Oct 30 12:44:19.405037 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.405019 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wkzj5" Oct 30 12:44:19.405820 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:19.405740 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-194.ec2.internal.18e34f4aceef1ea7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-194.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-142-194.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-142-194.ec2.internal,},FirstTimestamp:2026-10-30 12:44:19.341541031 +0000 UTC m=+0.441861842,LastTimestamp:2026-10-30 12:44:19.341541031 +0000 UTC m=+0.441861842,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-194.ec2.internal,}" Oct 30 12:44:19.422004 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:19.421985 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Oct 30 12:44:19.422085 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:19.422007 2574 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-142-194.ec2.internal.18e34f4aceee8f3f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-194.ec2.internal.18e34f4aceee8f3f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-194.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-194.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-194.ec2.internal,},FirstTimestamp:2026-10-30 12:44:19.341504319 +0000 UTC m=+0.441825125,LastTimestamp:2026-10-30 12:44:19.342911468 +0000 UTC m=+0.443232284,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-194.ec2.internal,}" Oct 30 12:44:19.481110 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.481007 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:44:19.482315 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.482297 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:44:19.482428 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.482329 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:44:19.482428 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.482340 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:44:19.482428 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.482369 2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-194.ec2.internal" Oct 30 12:44:19.488479 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.488461 2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-194.ec2.internal"] Oct 30 12:44:19.488527 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.488519 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:44:19.489442 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.489426 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:44:19.489517 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.489452 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:44:19.489517 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.489463 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:44:19.491788 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.491770 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:44:19.491948 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.491931 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" Oct 30 12:44:19.491985 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.491969 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:44:19.492543 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.492526 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:44:19.492637 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.492532 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:44:19.492637 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.492580 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:44:19.492637 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.492592 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:44:19.492637 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.492558 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:44:19.492637 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.492634 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:44:19.494741 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.494725 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-194.ec2.internal" Oct 30 12:44:19.494835 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.494754 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:44:19.495565 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.495537 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:44:19.495636 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.495569 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:44:19.495636 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.495584 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:44:19.503199 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.503184 2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-194.ec2.internal" Oct 30 12:44:19.503276 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:19.503208 2574 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-194.ec2.internal\": node \"ip-10-0-142-194.ec2.internal\" not found" Oct 30 12:44:19.513665 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:19.513644 2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-194.ec2.internal\" not found" node="ip-10-0-142-194.ec2.internal" Oct 30 12:44:19.515889 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:19.515874 2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-194.ec2.internal\" not found" node="ip-10-0-142-194.ec2.internal" Oct 30 12:44:19.521514 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.521497 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7bc488b7d1d453086a471da9a46ef80c-config\") pod \"kube-apiserver-proxy-ip-10-0-142-194.ec2.internal\" (UID: \"7bc488b7d1d453086a471da9a46ef80c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-194.ec2.internal" Oct 30 12:44:19.521578 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.521520 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fba8e851cd321bfcfae1acf3abfda6d3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal\" (UID: \"fba8e851cd321bfcfae1acf3abfda6d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" Oct 30 12:44:19.521578 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.521542 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fba8e851cd321bfcfae1acf3abfda6d3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal\" (UID: \"fba8e851cd321bfcfae1acf3abfda6d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" Oct 30 12:44:19.543025 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:19.543008 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-194.ec2.internal\" not found" Oct 30 12:44:19.621696 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.621667 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fba8e851cd321bfcfae1acf3abfda6d3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal\" (UID: \"fba8e851cd321bfcfae1acf3abfda6d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" Oct 30 12:44:19.621813 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.621707 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7bc488b7d1d453086a471da9a46ef80c-config\") pod \"kube-apiserver-proxy-ip-10-0-142-194.ec2.internal\" (UID: \"7bc488b7d1d453086a471da9a46ef80c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-194.ec2.internal" Oct 30 12:44:19.621813 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.621723 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fba8e851cd321bfcfae1acf3abfda6d3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal\" (UID: \"fba8e851cd321bfcfae1acf3abfda6d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" Oct 30 12:44:19.621813 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.621778 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fba8e851cd321bfcfae1acf3abfda6d3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal\" (UID: \"fba8e851cd321bfcfae1acf3abfda6d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" Oct 30 12:44:19.621813 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.621793 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fba8e851cd321bfcfae1acf3abfda6d3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal\" (UID: \"fba8e851cd321bfcfae1acf3abfda6d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" Oct 30 12:44:19.621970 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.621847 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7bc488b7d1d453086a471da9a46ef80c-config\") pod \"kube-apiserver-proxy-ip-10-0-142-194.ec2.internal\" (UID: \"7bc488b7d1d453086a471da9a46ef80c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-194.ec2.internal" Oct 30 12:44:19.643750 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:19.643730 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-194.ec2.internal\" not found" Oct 30 12:44:19.744307 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:19.744239 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-194.ec2.internal\" not found" Oct 30 12:44:19.818825 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.818800 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" Oct 30 12:44:19.822044 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:19.822025 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-194.ec2.internal" Oct 30 12:44:19.844744 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:19.844716 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-194.ec2.internal\" not found" Oct 30 12:44:19.945379 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:19.945341 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-194.ec2.internal\" not found" Oct 30 12:44:21.241079 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:21.240915 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-194.ec2.internal\" not found" Oct 30 12:44:21.341487 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:21.341460 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-194.ec2.internal\" not found" Oct 30 12:44:21.440991 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:21.440961 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 12:44:21.442034 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:21.442017 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-194.ec2.internal\" not found" Oct 30 12:44:21.543178 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:21.542943 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-194.ec2.internal\" not found" Oct 30 12:44:21.605916 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:21.605603 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 12:39:20 +0000 UTC" deadline="2028-04-12 18:33:50.729259094 +0000 UTC" Oct 30 12:44:21.605916 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:21.605638 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12725h49m29.123625318s" Oct 30 12:44:21.622872 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:21.622849 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:44:21.623850 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:21.623834 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:44:21.623930 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:21.623871 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:44:21.623930 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:21.623884 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:44:21.643355 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:21.643334 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-194.ec2.internal\" not found" Oct 30 12:44:21.681291 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:21.681254 2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:44:21.700700 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:21.700664 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bc488b7d1d453086a471da9a46ef80c.slice/crio-b6d636bf80c4c19da1db3aa6ee2fe68380d6a29b597a7941fcee2644cef8b287 WatchSource:0}: Error finding container b6d636bf80c4c19da1db3aa6ee2fe68380d6a29b597a7941fcee2644cef8b287: Status 404 returned error can't find the container with id b6d636bf80c4c19da1db3aa6ee2fe68380d6a29b597a7941fcee2644cef8b287 Oct 30 12:44:21.701086 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:21.701062 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba8e851cd321bfcfae1acf3abfda6d3.slice/crio-5e12ab82540a054fe54f452c4a8daa219be67c5c24fe14221a6d11d74b209d0f WatchSource:0}: Error finding container 5e12ab82540a054fe54f452c4a8daa219be67c5c24fe14221a6d11d74b209d0f: Status 404 returned error can't find the container with id 5e12ab82540a054fe54f452c4a8daa219be67c5c24fe14221a6d11d74b209d0f Oct 30 12:44:21.707662 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:21.707647 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:44:21.744172 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:21.744092 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-194.ec2.internal\" not found" Oct 30 12:44:21.844671 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:21.844590 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-194.ec2.internal\" not found" Oct 30 12:44:21.881869 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:21.881838 2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:44:21.913050 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:21.913026 2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:44:21.914037 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:21.914022 2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-194.ec2.internal" Oct 30 12:44:21.936391 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:21.936371 2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 12:44:21.937468 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:21.937454 2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" Oct 30 12:44:21.961558 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:21.961539 2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 12:44:22.030898 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.030869 2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:44:22.485703 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.485667 2574 apiserver.go:52] "Watching apiserver" Oct 30 12:44:22.514842 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.514817 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 12:44:22.526725 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.526701 2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 12:44:22.527131 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.527106 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-ljvqz","openshift-network-operator/iptables-alerter-9tp7b","openshift-ovn-kubernetes/ovnkube-node-8rpk8","kube-system/global-pull-secret-syncer-nk9fb","openshift-dns/node-resolver-gt8nx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal","openshift-multus/multus-r54rk","openshift-multus/network-metrics-daemon-j6b2w","kube-system/konnectivity-agent-k2fjm","openshift-cluster-node-tuning-operator/tuned-js5rp","kube-system/kube-apiserver-proxy-ip-10-0-142-194.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v","openshift-multus/multus-additional-cni-plugins-l2fgb","openshift-image-registry/node-ca-rjhg6"] Oct 30 12:44:22.533905 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.533881 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:44:22.536397 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.536093 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9tp7b" Oct 30 12:44:22.538589 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.538569 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.540818 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.540797 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rjhg6" Oct 30 12:44:22.543136 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.543115 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gt8nx" Oct 30 12:44:22.546033 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.546015 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:44:22.548167 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.548141 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljvqz" Oct 30 12:44:22.548260 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:22.548225 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljvqz" podUID="eb4ec7a3-2fd7-44dd-98ca-72710972a050" Oct 30 12:44:22.549245 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.549225 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 12:44:22.549364 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.549286 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 12:44:22.549429 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.549394 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 12:44:22.549535 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.549503 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mwttw\"" Oct 30 12:44:22.550638 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.550595 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 12:44:22.550768 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.550720 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:44:22.550963 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.550935 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6g9fh\"" Oct 30 12:44:22.551042 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.550995 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 12:44:22.553018 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.552583 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:44:22.553018 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:22.552656 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6b2w" podUID="7664727a-bcd2-4169-b70b-a0ecba045321" Oct 30 12:44:22.553193 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.553172 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 12:44:22.553320 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.553213 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 12:44:22.553399 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.553260 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pglr2\"" Oct 30 12:44:22.553399 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.553344 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 12:44:22.553399 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.553358 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 12:44:22.553541 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.553453 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 12:44:22.554619 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.554600 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k2fjm" Oct 30 12:44:22.557652 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.557633 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.560886 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.560869 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.562898 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.562881 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:44:22.562993 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:22.562949 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk9fb" podUID="21ea018f-70e6-4ee3-952d-704089532abb" Oct 30 12:44:22.564051 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.564026 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 12:44:22.564245 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.564230 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 12:44:22.564529 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.564493 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 12:44:22.564601 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.564555 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 12:44:22.564787 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.564767 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-svz4t\"" Oct 30 12:44:22.565890 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.565871 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-trr9l\"" Oct 30 12:44:22.567200 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.566477 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 12:44:22.567200 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.566601 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 12:44:22.567200 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.566735 2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 12:44:22.567200 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.566841 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2c8x8" Oct 30 12:44:22.567200 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.567042 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 12:44:22.567200 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.567040 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 12:44:22.567200 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.567134 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 12:44:22.567200 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.567170 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 12:44:22.567200 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.567187 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 12:44:22.567716 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.567240 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-k9vxw\"" Oct 30 12:44:22.575729 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.575705 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2c8x8" Oct 30 12:44:22.593993 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.593943 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" event={"ID":"fba8e851cd321bfcfae1acf3abfda6d3","Type":"ContainerStarted","Data":"5e12ab82540a054fe54f452c4a8daa219be67c5c24fe14221a6d11d74b209d0f"} Oct 30 12:44:22.596518 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.596490 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-194.ec2.internal" event={"ID":"7bc488b7d1d453086a471da9a46ef80c","Type":"ContainerStarted","Data":"b6d636bf80c4c19da1db3aa6ee2fe68380d6a29b597a7941fcee2644cef8b287"} Oct 30 12:44:22.617117 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.617093 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 12:44:22.629505 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.629478 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1a6ee59d-3b62-4be1-87e8-79bb3bece069-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:44:22.629631 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.629511 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7664727a-bcd2-4169-b70b-a0ecba045321-metrics-certs\") pod \"network-metrics-daemon-j6b2w\" (UID: \"7664727a-bcd2-4169-b70b-a0ecba045321\") " pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:44:22.629631 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.629536 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/37620bdd-11a4-4c12-8be1-9813a85d575b-agent-certs\") pod \"konnectivity-agent-k2fjm\" (UID: \"37620bdd-11a4-4c12-8be1-9813a85d575b\") " pod="kube-system/konnectivity-agent-k2fjm" Oct 30 12:44:22.629631 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.629573 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-etc-selinux\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:44:22.629631 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.629604 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-kubernetes\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.629836 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.629640 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-tuned\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.629836 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.629678 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-kubelet\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.629836 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.629732 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-run-ovn\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.629836 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.629753 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dkqx\" (UniqueName: \"kubernetes.io/projected/bc5eaee0-b01e-432b-86ae-0d97a04da650-kube-api-access-5dkqx\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.629836 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.629796 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4rqz\" (UniqueName: \"kubernetes.io/projected/eb4ec7a3-2fd7-44dd-98ca-72710972a050-kube-api-access-j4rqz\") pod \"network-check-target-ljvqz\" (UID: \"eb4ec7a3-2fd7-44dd-98ca-72710972a050\") " pod="openshift-network-diagnostics/network-check-target-ljvqz" Oct 30 12:44:22.629836 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.629817 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-run-netns\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.630068 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.629841 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-systemd\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.630068 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.629871 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-var-lib-kubelet\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.630068 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.629904 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:44:22.630068 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.629927 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-run-openvswitch\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.630068 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.629952 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21ea018f-70e6-4ee3-952d-704089532abb-kubelet-config\") pod \"global-pull-secret-syncer-nk9fb\" (UID: \"21ea018f-70e6-4ee3-952d-704089532abb\") " pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:44:22.630068 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.629975 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21ea018f-70e6-4ee3-952d-704089532abb-dbus\") pod \"global-pull-secret-syncer-nk9fb\" (UID: \"21ea018f-70e6-4ee3-952d-704089532abb\") " pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:44:22.630068 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630029 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a6ee59d-3b62-4be1-87e8-79bb3bece069-cnibin\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:44:22.630068 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630055 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a050566-8e52-4ea5-a4b6-575b1b4c9548-cni-binary-copy\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.630338 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630078 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-sysctl-d\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.630338 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630100 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-sysctl-conf\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.630338 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630124 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-lib-modules\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.630338 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630147 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49c39196-4e77-4bcf-86b1-705a263e394b-host\") pod \"node-ca-rjhg6\" (UID: \"49c39196-4e77-4bcf-86b1-705a263e394b\") " pod="openshift-image-registry/node-ca-rjhg6" Oct 30 12:44:22.630338 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630200 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a6ee59d-3b62-4be1-87e8-79bb3bece069-system-cni-dir\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:44:22.630338 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630225 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-system-cni-dir\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.630338 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630250 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-var-lib-kubelet\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.630338 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630286 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0-iptables-alerter-script\") pod \"iptables-alerter-9tp7b\" (UID: \"1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0\") " pod="openshift-network-operator/iptables-alerter-9tp7b" Oct 30 12:44:22.630338 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630313 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-device-dir\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:44:22.630664 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630344 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.630664 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630370 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbk56\" (UniqueName: \"kubernetes.io/projected/b2bdf343-b9cd-484a-a8c1-1d0713d74093-kube-api-access-cbk56\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.630664 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630393 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpmfr\" (UniqueName: \"kubernetes.io/projected/24a00317-8fb9-41f3-8e1a-d1d985887725-kube-api-access-lpmfr\") pod \"node-resolver-gt8nx\" (UID: \"24a00317-8fb9-41f3-8e1a-d1d985887725\") " pod="openshift-dns/node-resolver-gt8nx" Oct 30 12:44:22.630664 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630416 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-etc-openvswitch\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.630664 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630442 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bc5eaee0-b01e-432b-86ae-0d97a04da650-env-overrides\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.630664 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630477 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bc5eaee0-b01e-432b-86ae-0d97a04da650-ovnkube-script-lib\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.630664 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630500 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-multus-socket-dir-parent\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.630664 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630524 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-run-k8s-cni-cncf-io\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.630664 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630610 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6rnd\" (UniqueName: \"kubernetes.io/projected/8a050566-8e52-4ea5-a4b6-575b1b4c9548-kube-api-access-v6rnd\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.630664 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630647 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-sysconfig\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.630974 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630673 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0-host-slash\") pod \"iptables-alerter-9tp7b\" (UID: \"1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0\") " pod="openshift-network-operator/iptables-alerter-9tp7b" Oct 30 12:44:22.630974 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630698 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-cni-bin\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.630974 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630720 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/49c39196-4e77-4bcf-86b1-705a263e394b-serviceca\") pod \"node-ca-rjhg6\" (UID: \"49c39196-4e77-4bcf-86b1-705a263e394b\") " pod="openshift-image-registry/node-ca-rjhg6" Oct 30 12:44:22.630974 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630740 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2bdf343-b9cd-484a-a8c1-1d0713d74093-tmp\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.630974 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630754 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-multus-conf-dir\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.630974 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630770 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/24a00317-8fb9-41f3-8e1a-d1d985887725-hosts-file\") pod \"node-resolver-gt8nx\" (UID: \"24a00317-8fb9-41f3-8e1a-d1d985887725\") " pod="openshift-dns/node-resolver-gt8nx" Oct 30 12:44:22.630974 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630786 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-slash\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.630974 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630799 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-run-systemd\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.630974 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630815 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-run-ovn-kubernetes\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.630974 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630831 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bc5eaee0-b01e-432b-86ae-0d97a04da650-ovn-node-metrics-cert\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.630974 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630849 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-cnibin\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.630974 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630863 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-var-lib-cni-bin\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.630974 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630878 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-run-multus-certs\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.630974 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630897 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-sys-fs\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:44:22.630974 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630912 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h92pg\" (UniqueName: \"kubernetes.io/projected/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-kube-api-access-h92pg\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:44:22.630974 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630930 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-var-lib-openvswitch\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.630974 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630944 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-log-socket\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.631542 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630960 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-var-lib-cni-multus\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.631542 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.630984 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1a6ee59d-3b62-4be1-87e8-79bb3bece069-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:44:22.631542 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.631019 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs4g4\" (UniqueName: \"kubernetes.io/projected/1a6ee59d-3b62-4be1-87e8-79bb3bece069-kube-api-access-gs4g4\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:44:22.631542 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.631045 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/37620bdd-11a4-4c12-8be1-9813a85d575b-konnectivity-ca\") pod \"konnectivity-agent-k2fjm\" (UID: \"37620bdd-11a4-4c12-8be1-9813a85d575b\") " pod="kube-system/konnectivity-agent-k2fjm" Oct 30 12:44:22.631542 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.631063 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21ea018f-70e6-4ee3-952d-704089532abb-original-pull-secret\") pod \"global-pull-secret-syncer-nk9fb\" (UID: \"21ea018f-70e6-4ee3-952d-704089532abb\") " pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:44:22.631542 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.631080 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-systemd-units\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.631542 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.631186 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-node-log\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.631542 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.631203 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-cni-netd\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.631542 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.631220 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a6ee59d-3b62-4be1-87e8-79bb3bece069-cni-binary-copy\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:44:22.631542 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.631237 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8a050566-8e52-4ea5-a4b6-575b1b4c9548-multus-daemon-config\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.631542 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.631251 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-run\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.631542 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.631285 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn2wd\" (UniqueName: \"kubernetes.io/projected/49c39196-4e77-4bcf-86b1-705a263e394b-kube-api-access-sn2wd\") pod \"node-ca-rjhg6\" (UID: \"49c39196-4e77-4bcf-86b1-705a263e394b\") " pod="openshift-image-registry/node-ca-rjhg6" Oct 30 12:44:22.631542 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.631302 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzptg\" (UniqueName: \"kubernetes.io/projected/7664727a-bcd2-4169-b70b-a0ecba045321-kube-api-access-vzptg\") pod \"network-metrics-daemon-j6b2w\" (UID: \"7664727a-bcd2-4169-b70b-a0ecba045321\") " pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:44:22.631542 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.631316 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-sys\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.631542 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.631331 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-run-netns\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.631542 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.631375 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a6ee59d-3b62-4be1-87e8-79bb3bece069-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:44:22.632044 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.631391 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-etc-kubernetes\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.632044 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.631413 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-modprobe-d\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.632044 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.631436 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-host\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.632044 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.631454 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2szrp\" (UniqueName: \"kubernetes.io/projected/1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0-kube-api-access-2szrp\") pod \"iptables-alerter-9tp7b\" (UID: \"1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0\") " pod="openshift-network-operator/iptables-alerter-9tp7b" Oct 30 12:44:22.632044 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.631471 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24a00317-8fb9-41f3-8e1a-d1d985887725-tmp-dir\") pod \"node-resolver-gt8nx\" (UID: \"24a00317-8fb9-41f3-8e1a-d1d985887725\") " pod="openshift-dns/node-resolver-gt8nx" Oct 30 12:44:22.632044 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.631486 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-socket-dir\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:44:22.632044 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.631500 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-hostroot\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.632044 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.631524 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-multus-cni-dir\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.632044 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.631549 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-registration-dir\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:44:22.632044 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.631574 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bc5eaee0-b01e-432b-86ae-0d97a04da650-ovnkube-config\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.650106 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.650080 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 12:44:22.650228 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.650153 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 12:44:22.650228 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.650157 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:44:22.650228 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.650086 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 12:44:22.650370 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.650322 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 12:44:22.650370 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.650343 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-l246t\"" Oct 30 12:44:22.650442 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.650405 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6f4z2\"" Oct 30 12:44:22.650488 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.650450 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-b79n8\"" Oct 30 12:44:22.732596 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.732561 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-run-netns\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.732596 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.732596 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-systemd\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.732823 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.732624 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-var-lib-kubelet\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.732823 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.732649 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:44:22.732823 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.732673 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-run-openvswitch\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.732823 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.732672 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-run-netns\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.732823 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.732700 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21ea018f-70e6-4ee3-952d-704089532abb-kubelet-config\") pod \"global-pull-secret-syncer-nk9fb\" (UID: \"21ea018f-70e6-4ee3-952d-704089532abb\") " pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:44:22.732823 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.732724 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21ea018f-70e6-4ee3-952d-704089532abb-dbus\") pod \"global-pull-secret-syncer-nk9fb\" (UID: \"21ea018f-70e6-4ee3-952d-704089532abb\") " pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:44:22.732823 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.732747 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a6ee59d-3b62-4be1-87e8-79bb3bece069-cnibin\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:44:22.732823 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.732757 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-var-lib-kubelet\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.732823 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.732768 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a050566-8e52-4ea5-a4b6-575b1b4c9548-cni-binary-copy\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.732823 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.732786 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-sysctl-d\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.732823 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.732796 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-systemd\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.732823 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.732811 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-sysctl-conf\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.733206 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.732827 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-lib-modules\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.733206 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.732851 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49c39196-4e77-4bcf-86b1-705a263e394b-host\") pod \"node-ca-rjhg6\" (UID: \"49c39196-4e77-4bcf-86b1-705a263e394b\") " pod="openshift-image-registry/node-ca-rjhg6" Oct 30 12:44:22.733206 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.732874 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a6ee59d-3b62-4be1-87e8-79bb3bece069-system-cni-dir\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:44:22.733206 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.732898 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-system-cni-dir\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.733206 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.732917 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-var-lib-kubelet\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.733206 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.732932 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0-iptables-alerter-script\") pod \"iptables-alerter-9tp7b\" (UID: \"1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0\") " pod="openshift-network-operator/iptables-alerter-9tp7b" Oct 30 12:44:22.733206 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.732952 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-device-dir\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:44:22.733206 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.732969 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.733206 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.732974 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21ea018f-70e6-4ee3-952d-704089532abb-dbus\") pod \"global-pull-secret-syncer-nk9fb\" (UID: \"21ea018f-70e6-4ee3-952d-704089532abb\") " pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:44:22.733206 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.732985 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbk56\" (UniqueName: \"kubernetes.io/projected/b2bdf343-b9cd-484a-a8c1-1d0713d74093-kube-api-access-cbk56\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.733206 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.733013 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:44:22.733206 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.733021 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpmfr\" (UniqueName: \"kubernetes.io/projected/24a00317-8fb9-41f3-8e1a-d1d985887725-kube-api-access-lpmfr\") pod \"node-resolver-gt8nx\" (UID: \"24a00317-8fb9-41f3-8e1a-d1d985887725\") " pod="openshift-dns/node-resolver-gt8nx" Oct 30 12:44:22.733206 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.733038 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-etc-openvswitch\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.733206 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.733053 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bc5eaee0-b01e-432b-86ae-0d97a04da650-env-overrides\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.733206 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.733066 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21ea018f-70e6-4ee3-952d-704089532abb-kubelet-config\") pod \"global-pull-secret-syncer-nk9fb\" (UID: \"21ea018f-70e6-4ee3-952d-704089532abb\") " pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:44:22.733206 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.733068 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bc5eaee0-b01e-432b-86ae-0d97a04da650-ovnkube-script-lib\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.733206 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.733090 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-multus-socket-dir-parent\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.733789 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.733095 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-var-lib-kubelet\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.733789 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.733116 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-run-k8s-cni-cncf-io\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.733789 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.733130 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a6ee59d-3b62-4be1-87e8-79bb3bece069-cnibin\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:44:22.733789 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.733138 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6rnd\" (UniqueName: \"kubernetes.io/projected/8a050566-8e52-4ea5-a4b6-575b1b4c9548-kube-api-access-v6rnd\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.733789 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.733167 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-sysconfig\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.733789 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.733189 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0-host-slash\") pod \"iptables-alerter-9tp7b\" (UID: \"1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0\") " pod="openshift-network-operator/iptables-alerter-9tp7b" Oct 30 12:44:22.733789 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.733209 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-cni-bin\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.733789 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.733227 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/49c39196-4e77-4bcf-86b1-705a263e394b-serviceca\") pod \"node-ca-rjhg6\" (UID: \"49c39196-4e77-4bcf-86b1-705a263e394b\") " pod="openshift-image-registry/node-ca-rjhg6" Oct 30 12:44:22.733789 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.733241 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2bdf343-b9cd-484a-a8c1-1d0713d74093-tmp\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.733789 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.733256 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-multus-conf-dir\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.733789 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.733293 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/24a00317-8fb9-41f3-8e1a-d1d985887725-hosts-file\") pod \"node-resolver-gt8nx\" (UID: \"24a00317-8fb9-41f3-8e1a-d1d985887725\") " pod="openshift-dns/node-resolver-gt8nx" Oct 30 12:44:22.733789 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.733309 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-slash\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.733789 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.733341 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-run-systemd\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.733789 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.733367 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-run-ovn-kubernetes\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.733789 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.733384 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bc5eaee0-b01e-432b-86ae-0d97a04da650-ovn-node-metrics-cert\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.733789 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.733399 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-cnibin\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.733789 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.733435 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-cnibin\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.734389 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.733916 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0-iptables-alerter-script\") pod \"iptables-alerter-9tp7b\" (UID: \"1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0\") " pod="openshift-network-operator/iptables-alerter-9tp7b" Oct 30 12:44:22.734389 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.733935 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a050566-8e52-4ea5-a4b6-575b1b4c9548-cni-binary-copy\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.734389 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.733966 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-var-lib-cni-bin\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.734389 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.733988 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-run-multus-certs\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.734389 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734015 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-sys-fs\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:44:22.734389 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734039 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-sysctl-d\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.734389 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734040 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h92pg\" (UniqueName: \"kubernetes.io/projected/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-kube-api-access-h92pg\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:44:22.734389 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734083 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-var-lib-openvswitch\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.734389 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734101 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-log-socket\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.734389 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734117 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-var-lib-cni-multus\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.734389 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734141 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1a6ee59d-3b62-4be1-87e8-79bb3bece069-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:44:22.734389 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734164 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs4g4\" (UniqueName: \"kubernetes.io/projected/1a6ee59d-3b62-4be1-87e8-79bb3bece069-kube-api-access-gs4g4\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:44:22.734389 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734192 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/37620bdd-11a4-4c12-8be1-9813a85d575b-konnectivity-ca\") pod \"konnectivity-agent-k2fjm\" (UID: \"37620bdd-11a4-4c12-8be1-9813a85d575b\") " pod="kube-system/konnectivity-agent-k2fjm" Oct 30 12:44:22.734389 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734214 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21ea018f-70e6-4ee3-952d-704089532abb-original-pull-secret\") pod \"global-pull-secret-syncer-nk9fb\" (UID: \"21ea018f-70e6-4ee3-952d-704089532abb\") " pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:44:22.734389 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734231 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-systemd-units\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.734389 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734245 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-node-log\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.734389 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734260 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-cni-netd\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.734961 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734291 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.734961 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734299 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a6ee59d-3b62-4be1-87e8-79bb3bece069-cni-binary-copy\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:44:22.734961 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734317 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8a050566-8e52-4ea5-a4b6-575b1b4c9548-multus-daemon-config\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.734961 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734337 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-run\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.734961 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734354 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn2wd\" (UniqueName: \"kubernetes.io/projected/49c39196-4e77-4bcf-86b1-705a263e394b-kube-api-access-sn2wd\") pod \"node-ca-rjhg6\" (UID: \"49c39196-4e77-4bcf-86b1-705a263e394b\") " pod="openshift-image-registry/node-ca-rjhg6" Oct 30 12:44:22.734961 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734380 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzptg\" (UniqueName: \"kubernetes.io/projected/7664727a-bcd2-4169-b70b-a0ecba045321-kube-api-access-vzptg\") pod \"network-metrics-daemon-j6b2w\" (UID: \"7664727a-bcd2-4169-b70b-a0ecba045321\") " pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:44:22.734961 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734403 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-sys\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.734961 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734424 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-run-netns\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.734961 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734440 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a6ee59d-3b62-4be1-87e8-79bb3bece069-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:44:22.734961 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734464 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-etc-kubernetes\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.734961 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734480 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-modprobe-d\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.734961 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734493 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-host\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.734961 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734509 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2szrp\" (UniqueName: \"kubernetes.io/projected/1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0-kube-api-access-2szrp\") pod \"iptables-alerter-9tp7b\" (UID: \"1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0\") " pod="openshift-network-operator/iptables-alerter-9tp7b" Oct 30 12:44:22.734961 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734530 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24a00317-8fb9-41f3-8e1a-d1d985887725-tmp-dir\") pod \"node-resolver-gt8nx\" (UID: \"24a00317-8fb9-41f3-8e1a-d1d985887725\") " pod="openshift-dns/node-resolver-gt8nx" Oct 30 12:44:22.734961 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734546 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-socket-dir\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:44:22.734961 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734576 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-hostroot\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.734961 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734600 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-etc-openvswitch\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.735535 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734603 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-multus-cni-dir\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.735535 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734632 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-registration-dir\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:44:22.735535 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734645 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-multus-cni-dir\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.735535 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734649 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bc5eaee0-b01e-432b-86ae-0d97a04da650-ovnkube-config\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.735535 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734675 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1a6ee59d-3b62-4be1-87e8-79bb3bece069-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:44:22.735535 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734693 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7664727a-bcd2-4169-b70b-a0ecba045321-metrics-certs\") pod \"network-metrics-daemon-j6b2w\" (UID: \"7664727a-bcd2-4169-b70b-a0ecba045321\") " pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:44:22.735535 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734708 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/37620bdd-11a4-4c12-8be1-9813a85d575b-agent-certs\") pod \"konnectivity-agent-k2fjm\" (UID: \"37620bdd-11a4-4c12-8be1-9813a85d575b\") " pod="kube-system/konnectivity-agent-k2fjm" Oct 30 12:44:22.735535 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734732 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-etc-selinux\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:44:22.735535 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734759 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-kubernetes\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.735535 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734778 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-tuned\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.735535 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734792 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-kubelet\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.735535 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734807 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-run-ovn\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.735535 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734823 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dkqx\" (UniqueName: \"kubernetes.io/projected/bc5eaee0-b01e-432b-86ae-0d97a04da650-kube-api-access-5dkqx\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.735535 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734840 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4rqz\" (UniqueName: \"kubernetes.io/projected/eb4ec7a3-2fd7-44dd-98ca-72710972a050-kube-api-access-j4rqz\") pod \"network-check-target-ljvqz\" (UID: \"eb4ec7a3-2fd7-44dd-98ca-72710972a050\") " pod="openshift-network-diagnostics/network-check-target-ljvqz" Oct 30 12:44:22.735535 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734864 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-node-log\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.735535 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.734978 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-lib-modules\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.735535 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.735022 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49c39196-4e77-4bcf-86b1-705a263e394b-host\") pod \"node-ca-rjhg6\" (UID: \"49c39196-4e77-4bcf-86b1-705a263e394b\") " pod="openshift-image-registry/node-ca-rjhg6" Oct 30 12:44:22.736138 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.735065 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a6ee59d-3b62-4be1-87e8-79bb3bece069-system-cni-dir\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:44:22.736138 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.735084 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-device-dir\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:44:22.736138 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.735134 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-cni-netd\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.736138 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.735153 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-system-cni-dir\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.736138 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.735200 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-sysconfig\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.736138 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.735236 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0-host-slash\") pod \"iptables-alerter-9tp7b\" (UID: \"1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0\") " pod="openshift-network-operator/iptables-alerter-9tp7b" Oct 30 12:44:22.736138 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.735251 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bc5eaee0-b01e-432b-86ae-0d97a04da650-ovnkube-config\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.736138 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.733039 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-run-openvswitch\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.736138 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.735293 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-cni-bin\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.736138 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.735655 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bc5eaee0-b01e-432b-86ae-0d97a04da650-env-overrides\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.736138 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.735670 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a6ee59d-3b62-4be1-87e8-79bb3bece069-cni-binary-copy\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:44:22.736138 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.735682 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/49c39196-4e77-4bcf-86b1-705a263e394b-serviceca\") pod \"node-ca-rjhg6\" (UID: \"49c39196-4e77-4bcf-86b1-705a263e394b\") " pod="openshift-image-registry/node-ca-rjhg6" Oct 30 12:44:22.736138 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.735947 2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 12:44:22.736772 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.736748 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-kubernetes\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.736842 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.736796 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-var-lib-cni-bin\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.736842 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.736806 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-run-k8s-cni-cncf-io\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.736905 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.736843 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-run-multus-certs\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.736905 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.736851 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-hostroot\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.736964 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.736898 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-sys-fs\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:44:22.736998 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.736960 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-registration-dir\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:44:22.737033 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.737013 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-run-ovn\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.737114 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:22.737094 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:44:22.737171 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.737160 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-sysctl-conf\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.737225 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:22.737189 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7664727a-bcd2-4169-b70b-a0ecba045321-metrics-certs podName:7664727a-bcd2-4169-b70b-a0ecba045321 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:23.237162749 +0000 UTC m=+3.142438269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7664727a-bcd2-4169-b70b-a0ecba045321-metrics-certs") pod "network-metrics-daemon-j6b2w" (UID: "7664727a-bcd2-4169-b70b-a0ecba045321") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:44:22.737514 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.737490 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-host\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.737648 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.737526 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-etc-selinux\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:44:22.737648 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.736756 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-multus-socket-dir-parent\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.737648 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.737537 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-etc-kubernetes\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.737797 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.737680 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-run\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.737797 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.737747 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bc5eaee0-b01e-432b-86ae-0d97a04da650-ovnkube-script-lib\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.738256 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.738023 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-socket-dir\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:44:22.738256 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.738041 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8a050566-8e52-4ea5-a4b6-575b1b4c9548-multus-daemon-config\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.738256 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.738104 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a6ee59d-3b62-4be1-87e8-79bb3bece069-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:44:22.738461 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.738317 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-run-netns\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.738461 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.738406 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-var-lib-openvswitch\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.738561 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.738474 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-multus-conf-dir\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.738561 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.738519 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/24a00317-8fb9-41f3-8e1a-d1d985887725-hosts-file\") pod \"node-resolver-gt8nx\" (UID: \"24a00317-8fb9-41f3-8e1a-d1d985887725\") " pod="openshift-dns/node-resolver-gt8nx" Oct 30 12:44:22.738561 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.738552 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-slash\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.738689 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.738591 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-run-systemd\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.738689 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.738591 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1a6ee59d-3b62-4be1-87e8-79bb3bece069-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:44:22.738689 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.738641 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-run-ovn-kubernetes\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.739048 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.738981 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1a6ee59d-3b62-4be1-87e8-79bb3bece069-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:44:22.739205 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.739078 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-modprobe-d\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.739205 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.739116 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-kubelet\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.739422 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.739200 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24a00317-8fb9-41f3-8e1a-d1d985887725-tmp-dir\") pod \"node-resolver-gt8nx\" (UID: \"24a00317-8fb9-41f3-8e1a-d1d985887725\") " pod="openshift-dns/node-resolver-gt8nx" Oct 30 12:44:22.739422 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.739217 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-sys\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.739422 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.739399 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-var-lib-cni-multus\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.739708 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:22.739429 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:44:22.739708 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.739447 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-systemd-units\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.739708 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.739448 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-log-socket\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.739708 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:22.739580 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21ea018f-70e6-4ee3-952d-704089532abb-original-pull-secret podName:21ea018f-70e6-4ee3-952d-704089532abb nodeName:}" failed. No retries permitted until 2026-10-30 12:44:23.23955491 +0000 UTC m=+3.144830429 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21ea018f-70e6-4ee3-952d-704089532abb-original-pull-secret") pod "global-pull-secret-syncer-nk9fb" (UID: "21ea018f-70e6-4ee3-952d-704089532abb") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:44:22.739987 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.739951 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/37620bdd-11a4-4c12-8be1-9813a85d575b-konnectivity-ca\") pod \"konnectivity-agent-k2fjm\" (UID: \"37620bdd-11a4-4c12-8be1-9813a85d575b\") " pod="kube-system/konnectivity-agent-k2fjm" Oct 30 12:44:22.741522 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.741499 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-tuned\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.742456 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.742432 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2bdf343-b9cd-484a-a8c1-1d0713d74093-tmp\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.742777 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.742756 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/37620bdd-11a4-4c12-8be1-9813a85d575b-agent-certs\") pod \"konnectivity-agent-k2fjm\" (UID: \"37620bdd-11a4-4c12-8be1-9813a85d575b\") " pod="kube-system/konnectivity-agent-k2fjm" Oct 30 12:44:22.743196 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.743169 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bc5eaee0-b01e-432b-86ae-0d97a04da650-ovn-node-metrics-cert\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.758034 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.757995 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpmfr\" (UniqueName: \"kubernetes.io/projected/24a00317-8fb9-41f3-8e1a-d1d985887725-kube-api-access-lpmfr\") pod \"node-resolver-gt8nx\" (UID: \"24a00317-8fb9-41f3-8e1a-d1d985887725\") " pod="openshift-dns/node-resolver-gt8nx" Oct 30 12:44:22.758217 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.758197 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbk56\" (UniqueName: \"kubernetes.io/projected/b2bdf343-b9cd-484a-a8c1-1d0713d74093-kube-api-access-cbk56\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.758330 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:22.758200 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:44:22.758330 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:22.758253 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:44:22.758330 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:22.758283 2574 projected.go:196] Error preparing data for projected volume kube-api-access-j4rqz for pod openshift-network-diagnostics/network-check-target-ljvqz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:44:22.758489 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:22.758347 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb4ec7a3-2fd7-44dd-98ca-72710972a050-kube-api-access-j4rqz podName:eb4ec7a3-2fd7-44dd-98ca-72710972a050 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:23.258333002 +0000 UTC m=+3.163608515 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-j4rqz" (UniqueName: "kubernetes.io/projected/eb4ec7a3-2fd7-44dd-98ca-72710972a050-kube-api-access-j4rqz") pod "network-check-target-ljvqz" (UID: "eb4ec7a3-2fd7-44dd-98ca-72710972a050") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:44:22.758655 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.758629 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6rnd\" (UniqueName: \"kubernetes.io/projected/8a050566-8e52-4ea5-a4b6-575b1b4c9548-kube-api-access-v6rnd\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.758716 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.758672 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h92pg\" (UniqueName: \"kubernetes.io/projected/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-kube-api-access-h92pg\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:44:22.762617 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.762592 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dkqx\" (UniqueName: \"kubernetes.io/projected/bc5eaee0-b01e-432b-86ae-0d97a04da650-kube-api-access-5dkqx\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.766540 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.766496 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzptg\" (UniqueName: \"kubernetes.io/projected/7664727a-bcd2-4169-b70b-a0ecba045321-kube-api-access-vzptg\") pod \"network-metrics-daemon-j6b2w\" (UID: \"7664727a-bcd2-4169-b70b-a0ecba045321\") " pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:44:22.766785 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.766762 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs4g4\" (UniqueName: \"kubernetes.io/projected/1a6ee59d-3b62-4be1-87e8-79bb3bece069-kube-api-access-gs4g4\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:44:22.767431 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.767390 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2szrp\" (UniqueName: \"kubernetes.io/projected/1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0-kube-api-access-2szrp\") pod \"iptables-alerter-9tp7b\" (UID: \"1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0\") " pod="openshift-network-operator/iptables-alerter-9tp7b" Oct 30 12:44:22.768212 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.768188 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn2wd\" (UniqueName: \"kubernetes.io/projected/49c39196-4e77-4bcf-86b1-705a263e394b-kube-api-access-sn2wd\") pod \"node-ca-rjhg6\" (UID: \"49c39196-4e77-4bcf-86b1-705a263e394b\") " pod="openshift-image-registry/node-ca-rjhg6" Oct 30 12:44:22.850967 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.850929 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:44:22.859451 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.859423 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9tp7b" Oct 30 12:44:22.862281 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:22.862236 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e40c7ec_9b38_4f24_8d7b_5689f029c0a3.slice/crio-7525ad76683e03ad57ea0c28ae15ecaf1c8b91efedf5c3c5076bb832aab1e008 WatchSource:0}: Error finding container 7525ad76683e03ad57ea0c28ae15ecaf1c8b91efedf5c3c5076bb832aab1e008: Status 404 returned error can't find the container with id 7525ad76683e03ad57ea0c28ae15ecaf1c8b91efedf5c3c5076bb832aab1e008 Oct 30 12:44:22.869403 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:22.869377 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1758dcf9_60ef_4d0c_9cb2_998bf3b61ff0.slice/crio-44d0e397fc4ff49927726cc881f7e74c7f036d0707f71c992e117b6f7a99ed79 WatchSource:0}: Error finding container 44d0e397fc4ff49927726cc881f7e74c7f036d0707f71c992e117b6f7a99ed79: Status 404 returned error can't find the container with id 44d0e397fc4ff49927726cc881f7e74c7f036d0707f71c992e117b6f7a99ed79 Oct 30 12:44:22.883495 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.883468 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rjhg6" Oct 30 12:44:22.885945 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.885921 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:22.888620 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.888598 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gt8nx" Oct 30 12:44:22.895309 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:22.895284 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c39196_4e77_4bcf_86b1_705a263e394b.slice/crio-d5af4ca60da37a49e6d2fb7625c05a2773f1bbecd91127113ce4562dd50da318 WatchSource:0}: Error finding container d5af4ca60da37a49e6d2fb7625c05a2773f1bbecd91127113ce4562dd50da318: Status 404 returned error can't find the container with id d5af4ca60da37a49e6d2fb7625c05a2773f1bbecd91127113ce4562dd50da318 Oct 30 12:44:22.896062 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:22.896042 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc5eaee0_b01e_432b_86ae_0d97a04da650.slice/crio-55b9ad4e2e989f2262058882b22a435d407b6cbde7e0eecc3efbb3662b5c036f WatchSource:0}: Error finding container 55b9ad4e2e989f2262058882b22a435d407b6cbde7e0eecc3efbb3662b5c036f: Status 404 returned error can't find the container with id 55b9ad4e2e989f2262058882b22a435d407b6cbde7e0eecc3efbb3662b5c036f Oct 30 12:44:22.901094 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:22.901068 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a00317_8fb9_41f3_8e1a_d1d985887725.slice/crio-508dfb09a0efe309cd0e4d6bad5cb15be4499a215e5928ba834d4e93372773d9 WatchSource:0}: Error finding container 508dfb09a0efe309cd0e4d6bad5cb15be4499a215e5928ba834d4e93372773d9: Status 404 returned error can't find the container with id 508dfb09a0efe309cd0e4d6bad5cb15be4499a215e5928ba834d4e93372773d9 Oct 30 12:44:22.905411 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.905392 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k2fjm" Oct 30 12:44:22.908556 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.908525 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:44:22.910514 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.910493 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r54rk" Oct 30 12:44:22.914093 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:22.913898 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:44:22.917223 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:22.917198 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37620bdd_11a4_4c12_8be1_9813a85d575b.slice/crio-e45d0bc7ec0cb1a81c9a35075a9a3ad86e1483cf5806ee6d6e1f6a57709967a7 WatchSource:0}: Error finding container e45d0bc7ec0cb1a81c9a35075a9a3ad86e1483cf5806ee6d6e1f6a57709967a7: Status 404 returned error can't find the container with id e45d0bc7ec0cb1a81c9a35075a9a3ad86e1483cf5806ee6d6e1f6a57709967a7 Oct 30 12:44:22.924804 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:22.924775 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a6ee59d_3b62_4be1_87e8_79bb3bece069.slice/crio-dad2b65cdfd7add6edfe6d49ac05b02f31d74f0d50ae71f853de7dc5771538e1 WatchSource:0}: Error finding container dad2b65cdfd7add6edfe6d49ac05b02f31d74f0d50ae71f853de7dc5771538e1: Status 404 returned error can't find the container with id dad2b65cdfd7add6edfe6d49ac05b02f31d74f0d50ae71f853de7dc5771538e1 Oct 30 12:44:23.238547 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:23.238514 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7664727a-bcd2-4169-b70b-a0ecba045321-metrics-certs\") pod \"network-metrics-daemon-j6b2w\" (UID: \"7664727a-bcd2-4169-b70b-a0ecba045321\") " pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:44:23.239116 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:23.238728 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:44:23.239116 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:23.238816 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7664727a-bcd2-4169-b70b-a0ecba045321-metrics-certs podName:7664727a-bcd2-4169-b70b-a0ecba045321 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:24.23878702 +0000 UTC m=+4.144062546 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7664727a-bcd2-4169-b70b-a0ecba045321-metrics-certs") pod "network-metrics-daemon-j6b2w" (UID: "7664727a-bcd2-4169-b70b-a0ecba045321") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:44:23.339441 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:23.339397 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21ea018f-70e6-4ee3-952d-704089532abb-original-pull-secret\") pod \"global-pull-secret-syncer-nk9fb\" (UID: \"21ea018f-70e6-4ee3-952d-704089532abb\") " pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:44:23.339529 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:23.339479 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4rqz\" (UniqueName: \"kubernetes.io/projected/eb4ec7a3-2fd7-44dd-98ca-72710972a050-kube-api-access-j4rqz\") pod \"network-check-target-ljvqz\" (UID: \"eb4ec7a3-2fd7-44dd-98ca-72710972a050\") " pod="openshift-network-diagnostics/network-check-target-ljvqz" Oct 30 12:44:23.339587 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:23.339525 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:44:23.339641 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:23.339591 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21ea018f-70e6-4ee3-952d-704089532abb-original-pull-secret podName:21ea018f-70e6-4ee3-952d-704089532abb nodeName:}" failed. No retries permitted until 2026-10-30 12:44:24.339573231 +0000 UTC m=+4.244848738 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21ea018f-70e6-4ee3-952d-704089532abb-original-pull-secret") pod "global-pull-secret-syncer-nk9fb" (UID: "21ea018f-70e6-4ee3-952d-704089532abb") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:44:23.339641 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:23.339601 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:44:23.339641 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:23.339618 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:44:23.339641 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:23.339630 2574 projected.go:196] Error preparing data for projected volume kube-api-access-j4rqz for pod openshift-network-diagnostics/network-check-target-ljvqz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:44:23.339834 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:23.339679 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb4ec7a3-2fd7-44dd-98ca-72710972a050-kube-api-access-j4rqz podName:eb4ec7a3-2fd7-44dd-98ca-72710972a050 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:24.339665586 +0000 UTC m=+4.244941111 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-j4rqz" (UniqueName: "kubernetes.io/projected/eb4ec7a3-2fd7-44dd-98ca-72710972a050-kube-api-access-j4rqz") pod "network-check-target-ljvqz" (UID: "eb4ec7a3-2fd7-44dd-98ca-72710972a050") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:44:23.578448 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:23.578343 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:39:22 +0000 UTC" deadline="2028-06-09 00:43:54.067093143 +0000 UTC" Oct 30 12:44:23.578448 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:23.578404 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14099h59m30.48869244s" Oct 30 12:44:23.604086 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:23.603928 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" event={"ID":"bc5eaee0-b01e-432b-86ae-0d97a04da650","Type":"ContainerStarted","Data":"55b9ad4e2e989f2262058882b22a435d407b6cbde7e0eecc3efbb3662b5c036f"} Oct 30 12:44:23.607490 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:23.607420 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9tp7b" event={"ID":"1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0","Type":"ContainerStarted","Data":"44d0e397fc4ff49927726cc881f7e74c7f036d0707f71c992e117b6f7a99ed79"} Oct 30 12:44:23.611304 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:23.611259 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" event={"ID":"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3","Type":"ContainerStarted","Data":"7525ad76683e03ad57ea0c28ae15ecaf1c8b91efedf5c3c5076bb832aab1e008"} Oct 30 12:44:23.615700 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:23.615673 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-194.ec2.internal" event={"ID":"7bc488b7d1d453086a471da9a46ef80c","Type":"ContainerStarted","Data":"fe9369a42fdc8549e6f5a0ea452c0730f33a4d478cb96fa84ed72cd3aec448b8"} Oct 30 12:44:23.630456 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:23.630363 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-js5rp" event={"ID":"b2bdf343-b9cd-484a-a8c1-1d0713d74093","Type":"ContainerStarted","Data":"581770e669b826b6b7ca0773165b7e12f8615c9c6f10c3cc39e7e274dac9eb6d"} Oct 30 12:44:23.634899 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:23.634872 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rjhg6" event={"ID":"49c39196-4e77-4bcf-86b1-705a263e394b","Type":"ContainerStarted","Data":"d5af4ca60da37a49e6d2fb7625c05a2773f1bbecd91127113ce4562dd50da318"} Oct 30 12:44:23.642387 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:23.641647 2574 generic.go:356] "Generic (PLEG): container finished" podID="fba8e851cd321bfcfae1acf3abfda6d3" containerID="801ca3f13c3f6c860263fe7b1e89db1f2fda7feba3e5c33407112880fed75c8d" exitCode=0 Oct 30 12:44:23.642387 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:23.642258 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" event={"ID":"fba8e851cd321bfcfae1acf3abfda6d3","Type":"ContainerDied","Data":"801ca3f13c3f6c860263fe7b1e89db1f2fda7feba3e5c33407112880fed75c8d"} Oct 30 12:44:23.647242 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:23.647210 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r54rk" event={"ID":"8a050566-8e52-4ea5-a4b6-575b1b4c9548","Type":"ContainerStarted","Data":"0ab47e3edb68621f3c234d249753548ecf5793b8b8f7e6979f58a69f451b244a"} Oct 30 12:44:23.661485 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:23.661414 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-194.ec2.internal" podStartSLOduration=2.661396677 podStartE2EDuration="2.661396677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:44:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:23.660256311 +0000 UTC m=+3.565531837" watchObservedRunningTime="2026-10-30 12:44:23.661396677 +0000 UTC m=+3.566672200" Oct 30 12:44:23.669672 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:23.669574 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l2fgb" event={"ID":"1a6ee59d-3b62-4be1-87e8-79bb3bece069","Type":"ContainerStarted","Data":"dad2b65cdfd7add6edfe6d49ac05b02f31d74f0d50ae71f853de7dc5771538e1"} Oct 30 12:44:23.671683 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:23.671454 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k2fjm" event={"ID":"37620bdd-11a4-4c12-8be1-9813a85d575b","Type":"ContainerStarted","Data":"e45d0bc7ec0cb1a81c9a35075a9a3ad86e1483cf5806ee6d6e1f6a57709967a7"} Oct 30 12:44:23.675963 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:23.675935 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gt8nx" event={"ID":"24a00317-8fb9-41f3-8e1a-d1d985887725","Type":"ContainerStarted","Data":"508dfb09a0efe309cd0e4d6bad5cb15be4499a215e5928ba834d4e93372773d9"} Oct 30 12:44:24.248878 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:24.248837 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7664727a-bcd2-4169-b70b-a0ecba045321-metrics-certs\") pod \"network-metrics-daemon-j6b2w\" (UID: \"7664727a-bcd2-4169-b70b-a0ecba045321\") " pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:44:24.249055 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:24.249019 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:44:24.249140 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:24.249081 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7664727a-bcd2-4169-b70b-a0ecba045321-metrics-certs podName:7664727a-bcd2-4169-b70b-a0ecba045321 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:26.249063735 +0000 UTC m=+6.154339251 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7664727a-bcd2-4169-b70b-a0ecba045321-metrics-certs") pod "network-metrics-daemon-j6b2w" (UID: "7664727a-bcd2-4169-b70b-a0ecba045321") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:44:24.350619 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:24.349790 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4rqz\" (UniqueName: \"kubernetes.io/projected/eb4ec7a3-2fd7-44dd-98ca-72710972a050-kube-api-access-j4rqz\") pod \"network-check-target-ljvqz\" (UID: \"eb4ec7a3-2fd7-44dd-98ca-72710972a050\") " pod="openshift-network-diagnostics/network-check-target-ljvqz" Oct 30 12:44:24.350619 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:24.349881 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21ea018f-70e6-4ee3-952d-704089532abb-original-pull-secret\") pod \"global-pull-secret-syncer-nk9fb\" (UID: \"21ea018f-70e6-4ee3-952d-704089532abb\") " pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:44:24.350619 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:24.350030 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:44:24.350619 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:24.350091 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21ea018f-70e6-4ee3-952d-704089532abb-original-pull-secret podName:21ea018f-70e6-4ee3-952d-704089532abb nodeName:}" failed. No retries permitted until 2026-10-30 12:44:26.350073958 +0000 UTC m=+6.255349461 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21ea018f-70e6-4ee3-952d-704089532abb-original-pull-secret") pod "global-pull-secret-syncer-nk9fb" (UID: "21ea018f-70e6-4ee3-952d-704089532abb") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:44:24.350619 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:24.350507 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:44:24.350619 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:24.350526 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:44:24.350619 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:24.350538 2574 projected.go:196] Error preparing data for projected volume kube-api-access-j4rqz for pod openshift-network-diagnostics/network-check-target-ljvqz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:44:24.350619 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:24.350581 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb4ec7a3-2fd7-44dd-98ca-72710972a050-kube-api-access-j4rqz podName:eb4ec7a3-2fd7-44dd-98ca-72710972a050 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:26.350567252 +0000 UTC m=+6.255842759 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-j4rqz" (UniqueName: "kubernetes.io/projected/eb4ec7a3-2fd7-44dd-98ca-72710972a050-kube-api-access-j4rqz") pod "network-check-target-ljvqz" (UID: "eb4ec7a3-2fd7-44dd-98ca-72710972a050") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:44:24.579145 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:24.579004 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:39:22 +0000 UTC" deadline="2028-03-30 01:00:21.058364262 +0000 UTC" Oct 30 12:44:24.579145 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:24.579064 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12396h15m56.479304187s" Oct 30 12:44:24.584205 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:24.583352 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljvqz" Oct 30 12:44:24.584205 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:24.583469 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljvqz" podUID="eb4ec7a3-2fd7-44dd-98ca-72710972a050" Oct 30 12:44:24.584205 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:24.583891 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:44:24.584205 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:24.583989 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk9fb" podUID="21ea018f-70e6-4ee3-952d-704089532abb" Oct 30 12:44:24.584205 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:24.584065 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:44:24.584205 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:24.584145 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6b2w" podUID="7664727a-bcd2-4169-b70b-a0ecba045321" Oct 30 12:44:24.693436 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:24.693323 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" event={"ID":"fba8e851cd321bfcfae1acf3abfda6d3","Type":"ContainerStarted","Data":"b9abd4dff098f60e20f324ee0fce69b015c3e10fb17c6dc0bcfe37658e02bb00"} Oct 30 12:44:24.735607 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:24.735545 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" podStartSLOduration=3.735525606 podStartE2EDuration="3.735525606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:44:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:24.734561611 +0000 UTC m=+4.639837135" watchObservedRunningTime="2026-10-30 12:44:24.735525606 +0000 UTC m=+4.640801130" Oct 30 12:44:26.264798 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:26.264758 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7664727a-bcd2-4169-b70b-a0ecba045321-metrics-certs\") pod \"network-metrics-daemon-j6b2w\" (UID: \"7664727a-bcd2-4169-b70b-a0ecba045321\") " pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:44:26.265304 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:26.264936 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:44:26.265304 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:26.265001 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7664727a-bcd2-4169-b70b-a0ecba045321-metrics-certs podName:7664727a-bcd2-4169-b70b-a0ecba045321 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:30.264983575 +0000 UTC m=+10.170259082 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7664727a-bcd2-4169-b70b-a0ecba045321-metrics-certs") pod "network-metrics-daemon-j6b2w" (UID: "7664727a-bcd2-4169-b70b-a0ecba045321") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:44:26.365830 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:26.365791 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21ea018f-70e6-4ee3-952d-704089532abb-original-pull-secret\") pod \"global-pull-secret-syncer-nk9fb\" (UID: \"21ea018f-70e6-4ee3-952d-704089532abb\") " pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:44:26.366006 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:26.365878 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4rqz\" (UniqueName: \"kubernetes.io/projected/eb4ec7a3-2fd7-44dd-98ca-72710972a050-kube-api-access-j4rqz\") pod \"network-check-target-ljvqz\" (UID: \"eb4ec7a3-2fd7-44dd-98ca-72710972a050\") " pod="openshift-network-diagnostics/network-check-target-ljvqz" Oct 30 12:44:26.366082 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:26.366010 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:44:26.366082 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:26.366030 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:44:26.366082 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:26.366043 2574 projected.go:196] Error preparing data for projected volume kube-api-access-j4rqz for pod openshift-network-diagnostics/network-check-target-ljvqz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:44:26.366227 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:26.366101 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb4ec7a3-2fd7-44dd-98ca-72710972a050-kube-api-access-j4rqz podName:eb4ec7a3-2fd7-44dd-98ca-72710972a050 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:30.366082613 +0000 UTC m=+10.271358121 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-j4rqz" (UniqueName: "kubernetes.io/projected/eb4ec7a3-2fd7-44dd-98ca-72710972a050-kube-api-access-j4rqz") pod "network-check-target-ljvqz" (UID: "eb4ec7a3-2fd7-44dd-98ca-72710972a050") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:44:26.366227 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:26.366168 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:44:26.366227 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:26.366200 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21ea018f-70e6-4ee3-952d-704089532abb-original-pull-secret podName:21ea018f-70e6-4ee3-952d-704089532abb nodeName:}" failed. No retries permitted until 2026-10-30 12:44:30.366190063 +0000 UTC m=+10.271465574 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21ea018f-70e6-4ee3-952d-704089532abb-original-pull-secret") pod "global-pull-secret-syncer-nk9fb" (UID: "21ea018f-70e6-4ee3-952d-704089532abb") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:44:26.585824 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:26.585745 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljvqz" Oct 30 12:44:26.585993 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:26.585895 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljvqz" podUID="eb4ec7a3-2fd7-44dd-98ca-72710972a050" Oct 30 12:44:26.586322 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:26.586299 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:44:26.586430 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:26.586413 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk9fb" podUID="21ea018f-70e6-4ee3-952d-704089532abb" Oct 30 12:44:26.586485 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:26.586469 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:44:26.586560 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:26.586541 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6b2w" podUID="7664727a-bcd2-4169-b70b-a0ecba045321" Oct 30 12:44:26.700095 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:26.700060 2574 generic.go:356] "Generic (PLEG): container finished" podID="1a6ee59d-3b62-4be1-87e8-79bb3bece069" containerID="4f8d43f2361effef849ec26f6315d3482590bede29bba8c92756fbee702e3ed4" exitCode=0 Oct 30 12:44:26.700205 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:26.700122 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l2fgb" event={"ID":"1a6ee59d-3b62-4be1-87e8-79bb3bece069","Type":"ContainerDied","Data":"4f8d43f2361effef849ec26f6315d3482590bede29bba8c92756fbee702e3ed4"} Oct 30 12:44:27.708289 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:27.708237 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k2fjm" event={"ID":"37620bdd-11a4-4c12-8be1-9813a85d575b","Type":"ContainerStarted","Data":"aaf9fc345a96416dd24699446952e614860fe986df2d428275289112abb853e0"} Oct 30 12:44:28.584422 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:28.584392 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:44:28.584582 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:28.584526 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk9fb" podUID="21ea018f-70e6-4ee3-952d-704089532abb" Oct 30 12:44:28.584688 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:28.584604 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:44:28.584741 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:28.584685 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6b2w" podUID="7664727a-bcd2-4169-b70b-a0ecba045321" Oct 30 12:44:28.584815 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:28.584739 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljvqz" Oct 30 12:44:28.584883 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:28.584810 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljvqz" podUID="eb4ec7a3-2fd7-44dd-98ca-72710972a050" Oct 30 12:44:28.715732 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:28.715620 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" event={"ID":"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3","Type":"ContainerStarted","Data":"4cff6eef8fbc20c8160742d60d52b566a9579f3daba64dab3aa56a76048722e7"} Oct 30 12:44:28.720396 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:28.720293 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rjhg6" event={"ID":"49c39196-4e77-4bcf-86b1-705a263e394b","Type":"ContainerStarted","Data":"4f919f1a640e4a60bc5322b862d206fb0e6178f72603db905db0d2b525c72aeb"} Oct 30 12:44:28.759942 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:28.759322 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-k2fjm" podStartSLOduration=4.993817263 podStartE2EDuration="8.759303628s" totalImagesPullingTime="3.765486365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:44:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:27.752813878 +0000 UTC m=+7.658089406" watchObservedRunningTime="2026-10-30 12:44:28.759303628 +0000 UTC m=+8.664579155" Oct 30 12:44:30.139773 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:30.139747 2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 12:44:30.301778 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:30.301685 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7664727a-bcd2-4169-b70b-a0ecba045321-metrics-certs\") pod \"network-metrics-daemon-j6b2w\" (UID: \"7664727a-bcd2-4169-b70b-a0ecba045321\") " pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:44:30.301933 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:30.301859 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:44:30.301990 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:30.301946 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7664727a-bcd2-4169-b70b-a0ecba045321-metrics-certs podName:7664727a-bcd2-4169-b70b-a0ecba045321 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:38.301924258 +0000 UTC m=+18.207199762 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7664727a-bcd2-4169-b70b-a0ecba045321-metrics-certs") pod "network-metrics-daemon-j6b2w" (UID: "7664727a-bcd2-4169-b70b-a0ecba045321") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:44:30.403533 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:30.402710 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21ea018f-70e6-4ee3-952d-704089532abb-original-pull-secret\") pod \"global-pull-secret-syncer-nk9fb\" (UID: \"21ea018f-70e6-4ee3-952d-704089532abb\") " pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:44:30.403533 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:30.402787 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4rqz\" (UniqueName: \"kubernetes.io/projected/eb4ec7a3-2fd7-44dd-98ca-72710972a050-kube-api-access-j4rqz\") pod \"network-check-target-ljvqz\" (UID: \"eb4ec7a3-2fd7-44dd-98ca-72710972a050\") " pod="openshift-network-diagnostics/network-check-target-ljvqz" Oct 30 12:44:30.403533 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:30.402946 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:44:30.403533 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:30.402964 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:44:30.403533 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:30.402977 2574 projected.go:196] Error preparing data for projected volume kube-api-access-j4rqz for pod openshift-network-diagnostics/network-check-target-ljvqz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:44:30.403533 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:30.403036 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb4ec7a3-2fd7-44dd-98ca-72710972a050-kube-api-access-j4rqz podName:eb4ec7a3-2fd7-44dd-98ca-72710972a050 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:38.403018647 +0000 UTC m=+18.308294162 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-j4rqz" (UniqueName: "kubernetes.io/projected/eb4ec7a3-2fd7-44dd-98ca-72710972a050-kube-api-access-j4rqz") pod "network-check-target-ljvqz" (UID: "eb4ec7a3-2fd7-44dd-98ca-72710972a050") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:44:30.403533 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:30.403453 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:44:30.403533 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:30.403506 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21ea018f-70e6-4ee3-952d-704089532abb-original-pull-secret podName:21ea018f-70e6-4ee3-952d-704089532abb nodeName:}" failed. No retries permitted until 2026-10-30 12:44:38.403488817 +0000 UTC m=+18.308764333 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21ea018f-70e6-4ee3-952d-704089532abb-original-pull-secret") pod "global-pull-secret-syncer-nk9fb" (UID: "21ea018f-70e6-4ee3-952d-704089532abb") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:44:30.581366 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:30.581190 2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T12:44:30.139771359Z","UUID":"1dc23a96-6bc5-4f20-aabc-b971ef809cc2","Handler":null,"Name":"","Endpoint":""} Oct 30 12:44:30.583610 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:30.583577 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:44:30.583739 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:30.583693 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk9fb" podUID="21ea018f-70e6-4ee3-952d-704089532abb" Oct 30 12:44:30.584128 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:30.584097 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:44:30.584239 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:30.584219 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6b2w" podUID="7664727a-bcd2-4169-b70b-a0ecba045321" Oct 30 12:44:30.584330 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:30.584303 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljvqz" Oct 30 12:44:30.584400 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:30.584380 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljvqz" podUID="eb4ec7a3-2fd7-44dd-98ca-72710972a050" Oct 30 12:44:30.584552 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:30.584535 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 12:44:30.584633 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:30.584564 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 12:44:30.728153 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:30.727493 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r54rk" event={"ID":"8a050566-8e52-4ea5-a4b6-575b1b4c9548","Type":"ContainerStarted","Data":"2984f3067c19b1ab5e2e2b5bd6a06d209130220160a7f992cb40dc958f376ed8"} Oct 30 12:44:30.730653 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:30.730586 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" event={"ID":"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3","Type":"ContainerStarted","Data":"4aaa708cffeab42231a53423459f9c5c57ea81453bd324e10edc03604f6db587"} Oct 30 12:44:30.772243 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:30.771302 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rjhg6" podStartSLOduration=5.991681177 podStartE2EDuration="10.771282016s" totalImagesPullingTime="4.779600839s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:44:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:28.759915085 +0000 UTC m=+8.665190609" watchObservedRunningTime="2026-10-30 12:44:30.771282016 +0000 UTC m=+10.676557540" Oct 30 12:44:30.772243 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:30.771883 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-r54rk" podStartSLOduration=2.291049795 podStartE2EDuration="9.771870618s" totalImagesPullingTime="7.480820823s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:44:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:30.770796973 +0000 UTC m=+10.676072496" watchObservedRunningTime="2026-10-30 12:44:30.771870618 +0000 UTC m=+10.677146141" Oct 30 12:44:30.776289 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:30.776117 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-k2fjm" Oct 30 12:44:30.777407 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:30.776706 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-k2fjm" Oct 30 12:44:31.735133 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:31.734908 2574 generic.go:356] "Generic (PLEG): container finished" podID="1a6ee59d-3b62-4be1-87e8-79bb3bece069" containerID="2ecd8df8a3c867cb553128cf9953f4255f2003deba3c9f7e6d086c2418f95ba5" exitCode=0 Oct 30 12:44:31.735581 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:31.735109 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l2fgb" event={"ID":"1a6ee59d-3b62-4be1-87e8-79bb3bece069","Type":"ContainerDied","Data":"2ecd8df8a3c867cb553128cf9953f4255f2003deba3c9f7e6d086c2418f95ba5"} Oct 30 12:44:31.737406 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:31.737367 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gt8nx" event={"ID":"24a00317-8fb9-41f3-8e1a-d1d985887725","Type":"ContainerStarted","Data":"3bca9f6484235a289322033afc0fccd7146a55df383a52ad795426f3224109a3"} Oct 30 12:44:31.742792 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:31.742222 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" event={"ID":"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3","Type":"ContainerStarted","Data":"a54fa7fe23d39a8f9f953494336a446db731245bee8d511de90b95f864204190"} Oct 30 12:44:31.866477 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:31.866415 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" podStartSLOduration=3.862081309 podStartE2EDuration="11.866393755s" totalImagesPullingTime="8.004312446s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:44:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:31.825508014 +0000 UTC m=+11.730783538" watchObservedRunningTime="2026-10-30 12:44:31.866393755 +0000 UTC m=+11.771669270" Oct 30 12:44:32.583001 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:32.582965 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:44:32.583001 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:32.583000 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:44:32.583227 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:32.582973 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljvqz" Oct 30 12:44:32.583227 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:32.583110 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6b2w" podUID="7664727a-bcd2-4169-b70b-a0ecba045321" Oct 30 12:44:32.583227 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:32.583180 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljvqz" podUID="eb4ec7a3-2fd7-44dd-98ca-72710972a050" Oct 30 12:44:32.583379 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:32.583289 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk9fb" podUID="21ea018f-70e6-4ee3-952d-704089532abb" Oct 30 12:44:32.745314 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:32.745260 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9tp7b" event={"ID":"1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0","Type":"ContainerStarted","Data":"7b26e54f651dc88c9cd22183fca80ef677885465fc577d9001179cdf6e3571ab"} Oct 30 12:44:32.786443 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:32.786383 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-gt8nx" podStartSLOduration=5.014877557 podStartE2EDuration="12.786363436s" totalImagesPullingTime="7.771485879s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:44:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:31.866909461 +0000 UTC m=+11.772184983" watchObservedRunningTime="2026-10-30 12:44:32.786363436 +0000 UTC m=+12.691638960" Oct 30 12:44:32.787167 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:32.787134 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9tp7b" podStartSLOduration=5.238852378 podStartE2EDuration="12.787125514s" totalImagesPullingTime="7.548273136s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:44:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:32.786153949 +0000 UTC m=+12.691429472" watchObservedRunningTime="2026-10-30 12:44:32.787125514 +0000 UTC m=+12.692401039" Oct 30 12:44:33.749343 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:33.749299 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-js5rp" event={"ID":"b2bdf343-b9cd-484a-a8c1-1d0713d74093","Type":"ContainerStarted","Data":"d69ce7219d24aa2493134d63cec143b73d43e3590c0d8c1ad5a29f0716649a43"} Oct 30 12:44:33.753087 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:33.753035 2574 generic.go:356] "Generic (PLEG): container finished" podID="1a6ee59d-3b62-4be1-87e8-79bb3bece069" containerID="8c35adbba6ffbc98b99b7318e5767509a09bdabebfa1d1d40f5b9451a6cdc473" exitCode=0 Oct 30 12:44:33.753689 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:33.753659 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l2fgb" event={"ID":"1a6ee59d-3b62-4be1-87e8-79bb3bece069","Type":"ContainerDied","Data":"8c35adbba6ffbc98b99b7318e5767509a09bdabebfa1d1d40f5b9451a6cdc473"} Oct 30 12:44:33.843966 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:33.843914 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-js5rp" podStartSLOduration=2.748250407 podStartE2EDuration="12.843900051s" totalImagesPullingTime="10.095649644s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:44:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:33.79551108 +0000 UTC m=+13.700786599" watchObservedRunningTime="2026-10-30 12:44:33.843900051 +0000 UTC m=+13.749175573" Oct 30 12:44:34.583628 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:34.583159 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljvqz" Oct 30 12:44:34.583628 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:34.583190 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:44:34.583628 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:34.583157 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:44:34.583628 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:34.583297 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljvqz" podUID="eb4ec7a3-2fd7-44dd-98ca-72710972a050" Oct 30 12:44:34.583628 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:34.583360 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6b2w" podUID="7664727a-bcd2-4169-b70b-a0ecba045321" Oct 30 12:44:34.583628 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:34.583455 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk9fb" podUID="21ea018f-70e6-4ee3-952d-704089532abb" Oct 30 12:44:34.758236 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:34.758196 2574 generic.go:356] "Generic (PLEG): container finished" podID="1a6ee59d-3b62-4be1-87e8-79bb3bece069" containerID="7c50afd1feaa4bdcf85dba3a5c549594e249274f4cf1b91c105e3a92dc22b2e4" exitCode=0 Oct 30 12:44:34.758958 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:34.758309 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l2fgb" event={"ID":"1a6ee59d-3b62-4be1-87e8-79bb3bece069","Type":"ContainerDied","Data":"7c50afd1feaa4bdcf85dba3a5c549594e249274f4cf1b91c105e3a92dc22b2e4"} Oct 30 12:44:36.582895 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:36.582709 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:44:36.583385 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:36.582709 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:44:36.583385 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:36.583006 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6b2w" podUID="7664727a-bcd2-4169-b70b-a0ecba045321" Oct 30 12:44:36.583385 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:36.582776 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljvqz" Oct 30 12:44:36.583385 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:36.583090 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk9fb" podUID="21ea018f-70e6-4ee3-952d-704089532abb" Oct 30 12:44:36.583385 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:36.583178 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljvqz" podUID="eb4ec7a3-2fd7-44dd-98ca-72710972a050" Oct 30 12:44:37.770149 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:37.770014 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" event={"ID":"bc5eaee0-b01e-432b-86ae-0d97a04da650","Type":"ContainerStarted","Data":"35b4cfd02f6d063ca01bbb5680a32af70d0deb3760a191615262c6c39ece1ca4"} Oct 30 12:44:37.770504 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:37.770231 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" event={"ID":"bc5eaee0-b01e-432b-86ae-0d97a04da650","Type":"ContainerStarted","Data":"0f0838102db3362f8714ec0344dc65705f0c7f8f0ec77022fb2237c610c538ac"} Oct 30 12:44:37.770504 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:37.770294 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" event={"ID":"bc5eaee0-b01e-432b-86ae-0d97a04da650","Type":"ContainerStarted","Data":"fc065b423619ad3324610680153df82091108f928f46079ab7682b5cd06b2231"} Oct 30 12:44:37.770504 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:37.770341 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" event={"ID":"bc5eaee0-b01e-432b-86ae-0d97a04da650","Type":"ContainerStarted","Data":"eb44305db236ceac6a79dccd73dacf77a81a806e5562f3484a933a1b19ba4abd"} Oct 30 12:44:37.772567 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:37.772545 2574 generic.go:356] "Generic (PLEG): container finished" podID="1a6ee59d-3b62-4be1-87e8-79bb3bece069" containerID="accefaf68319d46a349c7010819cdfd9934694c0d0dc8b9e70e55eec0636e46b" exitCode=0 Oct 30 12:44:37.772648 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:37.772576 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l2fgb" event={"ID":"1a6ee59d-3b62-4be1-87e8-79bb3bece069","Type":"ContainerDied","Data":"accefaf68319d46a349c7010819cdfd9934694c0d0dc8b9e70e55eec0636e46b"} Oct 30 12:44:38.365985 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:38.365954 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7664727a-bcd2-4169-b70b-a0ecba045321-metrics-certs\") pod \"network-metrics-daemon-j6b2w\" (UID: \"7664727a-bcd2-4169-b70b-a0ecba045321\") " pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:44:38.366128 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:38.366087 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:44:38.366168 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:38.366147 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7664727a-bcd2-4169-b70b-a0ecba045321-metrics-certs podName:7664727a-bcd2-4169-b70b-a0ecba045321 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:54.366132126 +0000 UTC m=+34.271407628 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7664727a-bcd2-4169-b70b-a0ecba045321-metrics-certs") pod "network-metrics-daemon-j6b2w" (UID: "7664727a-bcd2-4169-b70b-a0ecba045321") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:44:38.466680 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:38.466650 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21ea018f-70e6-4ee3-952d-704089532abb-original-pull-secret\") pod \"global-pull-secret-syncer-nk9fb\" (UID: \"21ea018f-70e6-4ee3-952d-704089532abb\") " pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:44:38.466844 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:38.466710 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4rqz\" (UniqueName: \"kubernetes.io/projected/eb4ec7a3-2fd7-44dd-98ca-72710972a050-kube-api-access-j4rqz\") pod \"network-check-target-ljvqz\" (UID: \"eb4ec7a3-2fd7-44dd-98ca-72710972a050\") " pod="openshift-network-diagnostics/network-check-target-ljvqz" Oct 30 12:44:38.466844 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:38.466780 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:44:38.466963 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:38.466835 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21ea018f-70e6-4ee3-952d-704089532abb-original-pull-secret podName:21ea018f-70e6-4ee3-952d-704089532abb nodeName:}" failed. No retries permitted until 2026-10-30 12:44:54.466821654 +0000 UTC m=+34.372097155 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21ea018f-70e6-4ee3-952d-704089532abb-original-pull-secret") pod "global-pull-secret-syncer-nk9fb" (UID: "21ea018f-70e6-4ee3-952d-704089532abb") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:44:38.466963 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:38.466860 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:44:38.466963 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:38.466881 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:44:38.466963 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:38.466892 2574 projected.go:196] Error preparing data for projected volume kube-api-access-j4rqz for pod openshift-network-diagnostics/network-check-target-ljvqz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:44:38.466963 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:38.466945 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb4ec7a3-2fd7-44dd-98ca-72710972a050-kube-api-access-j4rqz podName:eb4ec7a3-2fd7-44dd-98ca-72710972a050 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:54.46692992 +0000 UTC m=+34.372205434 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-j4rqz" (UniqueName: "kubernetes.io/projected/eb4ec7a3-2fd7-44dd-98ca-72710972a050-kube-api-access-j4rqz") pod "network-check-target-ljvqz" (UID: "eb4ec7a3-2fd7-44dd-98ca-72710972a050") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:44:38.583229 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:38.583203 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:44:38.583346 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:38.583329 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:44:38.583391 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:38.583345 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk9fb" podUID="21ea018f-70e6-4ee3-952d-704089532abb" Oct 30 12:44:38.583455 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:38.583406 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6b2w" podUID="7664727a-bcd2-4169-b70b-a0ecba045321" Oct 30 12:44:38.583455 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:38.583435 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljvqz" Oct 30 12:44:38.583594 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:38.583532 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljvqz" podUID="eb4ec7a3-2fd7-44dd-98ca-72710972a050" Oct 30 12:44:38.777635 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:38.777596 2574 generic.go:356] "Generic (PLEG): container finished" podID="1a6ee59d-3b62-4be1-87e8-79bb3bece069" containerID="2ed953f87fcd3b80b810eb9dcde021018945996eeab2216505eac23135e243ed" exitCode=0 Oct 30 12:44:38.778113 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:38.777679 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l2fgb" event={"ID":"1a6ee59d-3b62-4be1-87e8-79bb3bece069","Type":"ContainerDied","Data":"2ed953f87fcd3b80b810eb9dcde021018945996eeab2216505eac23135e243ed"} Oct 30 12:44:38.780633 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:38.780614 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" event={"ID":"bc5eaee0-b01e-432b-86ae-0d97a04da650","Type":"ContainerStarted","Data":"fc0da3b6a56e4bd5fa8f91eaa73b863d552fe28fcea530600ff925aa58d3fddc"} Oct 30 12:44:38.780728 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:38.780713 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" event={"ID":"bc5eaee0-b01e-432b-86ae-0d97a04da650","Type":"ContainerStarted","Data":"8be9444367da82e0476bc4151a6f6f6176f539faeec7253860466dd7637ebd4b"} Oct 30 12:44:39.785681 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:39.785648 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l2fgb" event={"ID":"1a6ee59d-3b62-4be1-87e8-79bb3bece069","Type":"ContainerStarted","Data":"2adaa4c678c1f3ec9782a3471bec7b7631682825070b6e78febfb6e08810fdb6"} Oct 30 12:44:39.857365 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:39.857325 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-l2fgb" podStartSLOduration=9.37357846 podStartE2EDuration="19.8573136s" totalImagesPullingTime="10.48373514s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:44:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:39.857205655 +0000 UTC m=+19.762481179" watchObservedRunningTime="2026-10-30 12:44:39.8573136 +0000 UTC m=+19.762589122" Oct 30 12:44:40.583382 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:40.583350 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljvqz" Oct 30 12:44:40.583550 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:40.583425 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljvqz" podUID="eb4ec7a3-2fd7-44dd-98ca-72710972a050" Oct 30 12:44:40.583550 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:40.583532 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:44:40.583671 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:40.583655 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:44:40.583741 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:40.583652 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6b2w" podUID="7664727a-bcd2-4169-b70b-a0ecba045321" Oct 30 12:44:40.583788 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:40.583749 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk9fb" podUID="21ea018f-70e6-4ee3-952d-704089532abb" Oct 30 12:44:40.792464 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:40.792432 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" event={"ID":"bc5eaee0-b01e-432b-86ae-0d97a04da650","Type":"ContainerStarted","Data":"058d150c027a22a6efc252d63fc9dc3f57e6f6a2dc837069061714dc67cea8d3"} Oct 30 12:44:42.583149 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:42.582894 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:44:42.583815 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:42.582897 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljvqz" Oct 30 12:44:42.583815 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:42.582909 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:44:42.583815 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:42.583321 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljvqz" podUID="eb4ec7a3-2fd7-44dd-98ca-72710972a050" Oct 30 12:44:42.583815 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:42.583296 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk9fb" podUID="21ea018f-70e6-4ee3-952d-704089532abb" Oct 30 12:44:42.583815 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:42.583392 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6b2w" podUID="7664727a-bcd2-4169-b70b-a0ecba045321" Oct 30 12:44:42.799737 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:42.799701 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" event={"ID":"bc5eaee0-b01e-432b-86ae-0d97a04da650","Type":"ContainerStarted","Data":"2f360d806cb0486eda472a65e1109e3e825de180e269f4b05ee43e01941b9450"} Oct 30 12:44:42.800085 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:42.800058 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:42.800209 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:42.800133 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:42.816308 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:42.816283 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:42.890826 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:42.890709 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" podStartSLOduration=8.237617179 podStartE2EDuration="22.890697425s" totalImagesPullingTime="14.653080246s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:44:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:42.878005227 +0000 UTC m=+22.783280747" watchObservedRunningTime="2026-10-30 12:44:42.890697425 +0000 UTC m=+22.795972948" Oct 30 12:44:43.802260 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:43.802233 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:43.816243 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:43.816219 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:44:44.094075 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:44.094006 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nk9fb"] Oct 30 12:44:44.094201 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:44.094151 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:44:44.094255 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:44.094237 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk9fb" podUID="21ea018f-70e6-4ee3-952d-704089532abb" Oct 30 12:44:44.096986 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:44.096961 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ljvqz"] Oct 30 12:44:44.097100 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:44.097053 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljvqz" Oct 30 12:44:44.097157 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:44.097121 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljvqz" podUID="eb4ec7a3-2fd7-44dd-98ca-72710972a050" Oct 30 12:44:44.099992 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:44.099967 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-j6b2w"] Oct 30 12:44:44.100116 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:44.100078 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:44:44.100202 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:44.100182 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6b2w" podUID="7664727a-bcd2-4169-b70b-a0ecba045321" Oct 30 12:44:44.188437 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:44.188415 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-k2fjm" Oct 30 12:44:44.188637 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:44.188507 2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 12:44:44.188980 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:44.188962 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-k2fjm" Oct 30 12:44:45.582837 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:45.582805 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljvqz" Oct 30 12:44:45.583335 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:45.582805 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:44:45.583335 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:45.582926 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljvqz" podUID="eb4ec7a3-2fd7-44dd-98ca-72710972a050" Oct 30 12:44:45.583335 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:45.582805 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:44:45.583335 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:45.582982 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6b2w" podUID="7664727a-bcd2-4169-b70b-a0ecba045321" Oct 30 12:44:45.583335 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:45.583045 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk9fb" podUID="21ea018f-70e6-4ee3-952d-704089532abb" Oct 30 12:44:47.582639 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:47.582602 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:44:47.582639 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:47.582626 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:44:47.583184 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:47.582617 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljvqz" Oct 30 12:44:47.583184 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:47.582717 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk9fb" podUID="21ea018f-70e6-4ee3-952d-704089532abb" Oct 30 12:44:47.583184 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:47.582788 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljvqz" podUID="eb4ec7a3-2fd7-44dd-98ca-72710972a050" Oct 30 12:44:47.583184 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:47.582853 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6b2w" podUID="7664727a-bcd2-4169-b70b-a0ecba045321" Oct 30 12:44:49.583038 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:49.583004 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:44:49.583038 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:49.583029 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:44:49.583491 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:49.583137 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6b2w" podUID="7664727a-bcd2-4169-b70b-a0ecba045321" Oct 30 12:44:49.583491 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:49.583189 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljvqz" Oct 30 12:44:49.583491 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:49.583289 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk9fb" podUID="21ea018f-70e6-4ee3-952d-704089532abb" Oct 30 12:44:49.583491 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:49.583377 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljvqz" podUID="eb4ec7a3-2fd7-44dd-98ca-72710972a050" Oct 30 12:44:49.918100 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:49.918018 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeReady" Oct 30 12:44:49.918244 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:49.918159 2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 12:44:49.970218 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:49.970193 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-rnflh"] Oct 30 12:44:49.973517 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:49.973491 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cv78r"] Oct 30 12:44:49.973746 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:49.973621 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-rnflh" Oct 30 12:44:49.977052 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:49.977034 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6xnwh"] Oct 30 12:44:49.977188 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:49.977167 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-cv78r" Oct 30 12:44:49.979870 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:49.979851 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-5bmsj"] Oct 30 12:44:49.979976 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:49.979961 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6xnwh" Oct 30 12:44:49.982485 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:49.982464 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-kzhq7"] Oct 30 12:44:49.982569 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:49.982529 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-5bmsj" Oct 30 12:44:49.985340 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:49.985321 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5db587c4bc-dttsw"] Oct 30 12:44:49.986250 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:49.985592 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzhq7" Oct 30 12:44:49.988477 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:49.988458 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pptmg"] Oct 30 12:44:49.988579 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:49.988562 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:44:49.992259 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:49.992241 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-596f5cc4c5-sgvjw"] Oct 30 12:44:49.992354 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:49.992335 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pptmg" Oct 30 12:44:49.992505 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:49.992488 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 12:44:49.995007 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:49.994989 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-sztw6"] Oct 30 12:44:49.995105 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:49.995089 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:44:49.995197 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:49.995181 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 12:44:50.003882 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.003863 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-vjdmx"] Oct 30 12:44:50.004636 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.004375 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-sztw6" Oct 30 12:44:50.007315 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.007298 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7bff985b5d-qwb5l"] Oct 30 12:44:50.007409 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.007393 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjdmx" Oct 30 12:44:50.009983 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.009969 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-pkpds"] Oct 30 12:44:50.010095 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.010076 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:44:50.012590 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.012575 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-c8pgj"] Oct 30 12:44:50.012686 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.012671 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-pkpds" Oct 30 12:44:50.014322 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.014286 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 12:44:50.015233 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.015213 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-s7c7s"] Oct 30 12:44:50.015355 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.015320 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-c8pgj" Oct 30 12:44:50.017109 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.017090 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-9zx7f\"" Oct 30 12:44:50.017871 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.017847 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-v4mks"] Oct 30 12:44:50.017959 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.017942 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-s7c7s" Oct 30 12:44:50.020889 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.020628 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-rnflh"] Oct 30 12:44:50.020889 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.020669 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cv78r"] Oct 30 12:44:50.020889 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.020695 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-w52p8"] Oct 30 12:44:50.020889 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.020827 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" Oct 30 12:44:50.023481 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.023465 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6xnwh"] Oct 30 12:44:50.023553 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.023491 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dnpgs"] Oct 30 12:44:50.023617 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.023581 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w52p8" Oct 30 12:44:50.026114 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.026097 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dnpgs" Oct 30 12:44:50.040960 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.040937 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-5bmsj"] Oct 30 12:44:50.041662 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.041639 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-kzhq7"] Oct 30 12:44:50.046526 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.046503 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-596f5cc4c5-sgvjw"] Oct 30 12:44:50.049835 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.049815 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pptmg"] Oct 30 12:44:50.051085 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.051061 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e4bf0a41-a4bd-47ad-93b1-088500499acb-serving-cert\") pod \"service-ca-operator-544d749c8c-kzhq7\" (UID: \"e4bf0a41-a4bd-47ad-93b1-088500499acb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzhq7" Oct 30 12:44:50.051180 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.051120 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/877cfa0b-b1c6-4455-b232-b1ca558b5084-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5bmsj\" (UID: \"877cfa0b-b1c6-4455-b232-b1ca558b5084\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5bmsj" Oct 30 12:44:50.051244 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.051184 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e4bf0a41-a4bd-47ad-93b1-088500499acb-config\") pod \"service-ca-operator-544d749c8c-kzhq7\" (UID: \"e4bf0a41-a4bd-47ad-93b1-088500499acb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzhq7" Oct 30 12:44:50.051316 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.051253 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcpmg\" (UniqueName: \"kubernetes.io/projected/e4bf0a41-a4bd-47ad-93b1-088500499acb-kube-api-access-lcpmg\") pod \"service-ca-operator-544d749c8c-kzhq7\" (UID: \"e4bf0a41-a4bd-47ad-93b1-088500499acb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzhq7" Oct 30 12:44:50.051533 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.051515 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf8zf\" (UniqueName: \"kubernetes.io/projected/a3928331-ce67-4c37-9c5d-63b0da642a0f-kube-api-access-vf8zf\") pod \"kas-connection-checker-685895c7c9-rnflh\" (UID: \"a3928331-ce67-4c37-9c5d-63b0da642a0f\") " pod="kube-system/kas-connection-checker-685895c7c9-rnflh" Oct 30 12:44:50.051696 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.051671 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnz9b\" (UniqueName: \"kubernetes.io/projected/ae9fe0c2-0d00-4891-8549-bc99f5373da9-kube-api-access-pnz9b\") pod \"kas-connection-checker-685895c7c9-cv78r\" (UID: \"ae9fe0c2-0d00-4891-8549-bc99f5373da9\") " pod="kube-system/kas-connection-checker-685895c7c9-cv78r" Oct 30 12:44:50.051792 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.051713 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/877cfa0b-b1c6-4455-b232-b1ca558b5084-nginx-conf\") pod \"networking-console-plugin-86c478bb88-5bmsj\" (UID: \"877cfa0b-b1c6-4455-b232-b1ca558b5084\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5bmsj" Oct 30 12:44:50.051792 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.051745 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbwtv\" (UniqueName: \"kubernetes.io/projected/9c377915-c344-4791-baf6-4b286b9c019b-kube-api-access-mbwtv\") pod \"kas-connection-checker-685895c7c9-6xnwh\" (UID: \"9c377915-c344-4791-baf6-4b286b9c019b\") " pod="kube-system/kas-connection-checker-685895c7c9-6xnwh" Oct 30 12:44:50.052243 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.052223 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-c8pgj"] Oct 30 12:44:50.068388 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.068348 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-vjdmx"] Oct 30 12:44:50.068927 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.068910 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dnpgs"] Oct 30 12:44:50.071754 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.071730 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-s7c7s"] Oct 30 12:44:50.073779 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.073758 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5db587c4bc-dttsw"] Oct 30 12:44:50.079117 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.079099 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-pkpds"] Oct 30 12:44:50.079890 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.079859 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-v4mks"] Oct 30 12:44:50.083335 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.083316 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-sztw6"] Oct 30 12:44:50.088131 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.088110 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bff985b5d-qwb5l"] Oct 30 12:44:50.093437 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.093379 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w52p8"] Oct 30 12:44:50.096120 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.096095 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:44:50.098258 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.098236 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 12:44:50.099028 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.098260 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:44:50.099028 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.098417 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 12:44:50.099028 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.098630 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 12:44:50.099028 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.098868 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 12:44:50.099028 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.098907 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:44:50.099392 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.099061 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 12:44:50.099465 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.099441 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 12:44:50.099567 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.099554 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 12:44:50.099765 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.099746 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:44:50.099838 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.099766 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 12:44:50.100837 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.100158 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 12:44:50.100939 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.100293 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 12:44:50.101031 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.100347 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 12:44:50.101031 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.098261 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 12:44:50.101031 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.100440 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 12:44:50.101384 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.101362 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 12:44:50.101473 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.101447 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 12:44:50.101548 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.101473 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 12:44:50.101548 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.101534 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 12:44:50.104226 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.104210 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 12:44:50.104226 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.104212 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 12:44:50.107682 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.107657 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 12:44:50.107761 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.107734 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:44:50.107799 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.107790 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 12:44:50.107872 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.107839 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 12:44:50.113454 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.113439 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 12:44:50.121077 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.121030 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 12:44:50.121392 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.121372 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 12:44:50.121487 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.121457 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 12:44:50.121562 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.121505 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 12:44:50.121562 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.121540 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 12:44:50.121562 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.121564 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 12:44:50.121755 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.121594 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 12:44:50.121755 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.121596 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 12:44:50.121755 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.121626 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 12:44:50.121755 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.121644 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 12:44:50.121755 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.121654 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 12:44:50.121755 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.121661 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 12:44:50.121755 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.121668 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 12:44:50.121755 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.121684 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 12:44:50.121755 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.121671 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 12:44:50.121755 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.121692 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 12:44:50.121755 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.121716 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 12:44:50.121755 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.121719 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 12:44:50.121755 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.121730 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mxwzk\"" Oct 30 12:44:50.121755 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.121760 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xlfj2\"" Oct 30 12:44:50.122171 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.121805 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8b8c8\"" Oct 30 12:44:50.152806 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.152783 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfff9\" (UniqueName: \"kubernetes.io/projected/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-kube-api-access-wfff9\") pod \"insights-operator-7c4986d449-sztw6\" (UID: \"73b3e3a7-93fa-49c2-a8f1-4188a22b99a5\") " pod="openshift-insights/insights-operator-7c4986d449-sztw6" Oct 30 12:44:50.152888 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.152809 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkdqk\" (UniqueName: \"kubernetes.io/projected/4df70e16-5acb-494e-ab67-855c809e3411-kube-api-access-hkdqk\") pod \"ingress-canary-dnpgs\" (UID: \"4df70e16-5acb-494e-ab67-855c809e3411\") " pod="openshift-ingress-canary/ingress-canary-dnpgs" Oct 30 12:44:50.152888 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.152825 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-tls\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:44:50.152888 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.152843 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-default-certificate\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:44:50.152888 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.152861 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-installation-pull-secrets\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:44:50.153008 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.152921 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/71e82393-029b-4388-86df-8bd068720867-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pptmg\" (UID: \"71e82393-029b-4388-86df-8bd068720867\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pptmg" Oct 30 12:44:50.153008 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.152946 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cb39685-b929-48bd-afd2-228862e62668-tmp-dir\") pod \"dns-default-w52p8\" (UID: \"7cb39685-b929-48bd-afd2-228862e62668\") " pod="openshift-dns/dns-default-w52p8" Oct 30 12:44:50.153008 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.152980 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbnzh\" (UniqueName: \"kubernetes.io/projected/71e82393-029b-4388-86df-8bd068720867-kube-api-access-kbnzh\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pptmg\" (UID: \"71e82393-029b-4388-86df-8bd068720867\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pptmg" Oct 30 12:44:50.153008 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.153003 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d06e291-b9d3-48f7-8fc2-f615440631d9-service-ca-bundle\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:44:50.153122 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.153040 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g248\" (UniqueName: \"kubernetes.io/projected/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-kube-api-access-4g248\") pod \"cluster-samples-operator-6ddb6c8b94-s7c7s\" (UID: \"4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-s7c7s" Oct 30 12:44:50.153122 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.153069 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpcwv\" (UniqueName: \"kubernetes.io/projected/8107811a-ec7c-497a-83fc-372249074496-kube-api-access-jpcwv\") pod \"network-check-source-6f78d59f99-pkpds\" (UID: \"8107811a-ec7c-497a-83fc-372249074496\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-pkpds" Oct 30 12:44:50.153122 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.153088 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/00ee6aae-7264-4c8d-a337-c7c9f1132e9c-config\") pod \"console-operator-7cd56b8dbc-v4mks\" (UID: \"00ee6aae-7264-4c8d-a337-c7c9f1132e9c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" Oct 30 12:44:50.153203 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.153139 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnz9b\" (UniqueName: \"kubernetes.io/projected/ae9fe0c2-0d00-4891-8549-bc99f5373da9-kube-api-access-pnz9b\") pod \"kas-connection-checker-685895c7c9-cv78r\" (UID: \"ae9fe0c2-0d00-4891-8549-bc99f5373da9\") " pod="kube-system/kas-connection-checker-685895c7c9-cv78r" Oct 30 12:44:50.153203 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.153179 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wppl9\" (UniqueName: \"kubernetes.io/projected/64bdbc16-e845-45da-8f34-f0d788a75eac-kube-api-access-wppl9\") pod \"cluster-monitoring-operator-884576c57-vjdmx\" (UID: \"64bdbc16-e845-45da-8f34-f0d788a75eac\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjdmx" Oct 30 12:44:50.153259 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.153218 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-ca-trust-extracted\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:44:50.153345 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.153332 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-ca-trust-extracted\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:44:50.153383 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.153355 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4df70e16-5acb-494e-ab67-855c809e3411-cert\") pod \"ingress-canary-dnpgs\" (UID: \"4df70e16-5acb-494e-ab67-855c809e3411\") " pod="openshift-ingress-canary/ingress-canary-dnpgs" Oct 30 12:44:50.153417 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.153380 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6xkw\" (UniqueName: \"kubernetes.io/projected/7cb39685-b929-48bd-afd2-228862e62668-kube-api-access-p6xkw\") pod \"dns-default-w52p8\" (UID: \"7cb39685-b929-48bd-afd2-228862e62668\") " pod="openshift-dns/dns-default-w52p8" Oct 30 12:44:50.153450 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.153425 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-bound-sa-token\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:44:50.153481 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.153468 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7cb39685-b929-48bd-afd2-228862e62668-config-volume\") pod \"dns-default-w52p8\" (UID: \"7cb39685-b929-48bd-afd2-228862e62668\") " pod="openshift-dns/dns-default-w52p8" Oct 30 12:44:50.153511 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.153488 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-bound-sa-token\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:44:50.153542 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.153514 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbwtv\" (UniqueName: \"kubernetes.io/projected/9c377915-c344-4791-baf6-4b286b9c019b-kube-api-access-mbwtv\") pod \"kas-connection-checker-685895c7c9-6xnwh\" (UID: \"9c377915-c344-4791-baf6-4b286b9c019b\") " pod="kube-system/kas-connection-checker-685895c7c9-6xnwh" Oct 30 12:44:50.153573 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.153548 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e4bf0a41-a4bd-47ad-93b1-088500499acb-serving-cert\") pod \"service-ca-operator-544d749c8c-kzhq7\" (UID: \"e4bf0a41-a4bd-47ad-93b1-088500499acb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzhq7" Oct 30 12:44:50.153604 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.153571 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcpmg\" (UniqueName: \"kubernetes.io/projected/e4bf0a41-a4bd-47ad-93b1-088500499acb-kube-api-access-lcpmg\") pod \"service-ca-operator-544d749c8c-kzhq7\" (UID: \"e4bf0a41-a4bd-47ad-93b1-088500499acb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzhq7" Oct 30 12:44:50.153604 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.153589 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/877cfa0b-b1c6-4455-b232-b1ca558b5084-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5bmsj\" (UID: \"877cfa0b-b1c6-4455-b232-b1ca558b5084\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5bmsj" Oct 30 12:44:50.153668 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.153606 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e4bf0a41-a4bd-47ad-93b1-088500499acb-config\") pod \"service-ca-operator-544d749c8c-kzhq7\" (UID: \"e4bf0a41-a4bd-47ad-93b1-088500499acb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzhq7" Oct 30 12:44:50.153668 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.153632 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8wc5\" (UniqueName: \"kubernetes.io/projected/93e1032c-06a8-4418-99b7-540d1a4f1232-kube-api-access-p8wc5\") pod \"volume-data-source-validator-58c9f5f46d-c8pgj\" (UID: \"93e1032c-06a8-4418-99b7-540d1a4f1232\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-c8pgj" Oct 30 12:44:50.153668 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.153655 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-tmp\") pod \"insights-operator-7c4986d449-sztw6\" (UID: \"73b3e3a7-93fa-49c2-a8f1-4188a22b99a5\") " pod="openshift-insights/insights-operator-7c4986d449-sztw6" Oct 30 12:44:50.153823 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:50.153688 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 12:44:50.153823 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.153725 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-certificates\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:44:50.153823 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:50.153744 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/877cfa0b-b1c6-4455-b232-b1ca558b5084-networking-console-plugin-cert podName:877cfa0b-b1c6-4455-b232-b1ca558b5084 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:50.65372609 +0000 UTC m=+30.559001612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/877cfa0b-b1c6-4455-b232-b1ca558b5084-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5bmsj" (UID: "877cfa0b-b1c6-4455-b232-b1ca558b5084") : secret "networking-console-plugin-cert" not found Oct 30 12:44:50.153823 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.153805 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/00ee6aae-7264-4c8d-a337-c7c9f1132e9c-serving-cert\") pod \"console-operator-7cd56b8dbc-v4mks\" (UID: \"00ee6aae-7264-4c8d-a337-c7c9f1132e9c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" Oct 30 12:44:50.154015 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.153841 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/64bdbc16-e845-45da-8f34-f0d788a75eac-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-vjdmx\" (UID: \"64bdbc16-e845-45da-8f34-f0d788a75eac\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjdmx" Oct 30 12:44:50.154015 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.153946 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/71e82393-029b-4388-86df-8bd068720867-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pptmg\" (UID: \"71e82393-029b-4388-86df-8bd068720867\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pptmg" Oct 30 12:44:50.154015 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.153966 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-s7c7s\" (UID: \"4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-s7c7s" Oct 30 12:44:50.154015 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.153998 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-trusted-ca\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:44:50.154203 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.154032 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-s7c7s\" (UID: \"4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-s7c7s" Oct 30 12:44:50.154203 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.154072 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-snapshots\") pod \"insights-operator-7c4986d449-sztw6\" (UID: \"73b3e3a7-93fa-49c2-a8f1-4188a22b99a5\") " pod="openshift-insights/insights-operator-7c4986d449-sztw6" Oct 30 12:44:50.154203 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.154112 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf8zf\" (UniqueName: \"kubernetes.io/projected/a3928331-ce67-4c37-9c5d-63b0da642a0f-kube-api-access-vf8zf\") pod \"kas-connection-checker-685895c7c9-rnflh\" (UID: \"a3928331-ce67-4c37-9c5d-63b0da642a0f\") " pod="kube-system/kas-connection-checker-685895c7c9-rnflh" Oct 30 12:44:50.154203 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.154116 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e4bf0a41-a4bd-47ad-93b1-088500499acb-config\") pod \"service-ca-operator-544d749c8c-kzhq7\" (UID: \"e4bf0a41-a4bd-47ad-93b1-088500499acb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzhq7" Oct 30 12:44:50.154203 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.154170 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-certificates\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:44:50.154443 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.154226 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-installation-pull-secrets\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:44:50.154443 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.154313 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/877cfa0b-b1c6-4455-b232-b1ca558b5084-nginx-conf\") pod \"networking-console-plugin-86c478bb88-5bmsj\" (UID: \"877cfa0b-b1c6-4455-b232-b1ca558b5084\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5bmsj" Oct 30 12:44:50.154443 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.154338 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-image-registry-private-configuration\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:44:50.154443 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.154354 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zks6k\" (UniqueName: \"kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-kube-api-access-zks6k\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:44:50.154443 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.154376 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtcpz\" (UniqueName: \"kubernetes.io/projected/00ee6aae-7264-4c8d-a337-c7c9f1132e9c-kube-api-access-wtcpz\") pod \"console-operator-7cd56b8dbc-v4mks\" (UID: \"00ee6aae-7264-4c8d-a337-c7c9f1132e9c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" Oct 30 12:44:50.154681 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.154511 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-service-ca-bundle\") pod \"insights-operator-7c4986d449-sztw6\" (UID: \"73b3e3a7-93fa-49c2-a8f1-4188a22b99a5\") " pod="openshift-insights/insights-operator-7c4986d449-sztw6" Oct 30 12:44:50.154716 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.154702 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/64bdbc16-e845-45da-8f34-f0d788a75eac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vjdmx\" (UID: \"64bdbc16-e845-45da-8f34-f0d788a75eac\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjdmx" Oct 30 12:44:50.154748 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.154731 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-metrics-certs\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:44:50.154785 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.154753 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpzms\" (UniqueName: \"kubernetes.io/projected/1d06e291-b9d3-48f7-8fc2-f615440631d9-kube-api-access-jpzms\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:44:50.154785 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.154774 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00ee6aae-7264-4c8d-a337-c7c9f1132e9c-trusted-ca\") pod \"console-operator-7cd56b8dbc-v4mks\" (UID: \"00ee6aae-7264-4c8d-a337-c7c9f1132e9c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" Oct 30 12:44:50.154871 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.154810 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-trusted-ca\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:44:50.154916 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.154889 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-serving-cert\") pod \"insights-operator-7c4986d449-sztw6\" (UID: \"73b3e3a7-93fa-49c2-a8f1-4188a22b99a5\") " pod="openshift-insights/insights-operator-7c4986d449-sztw6" Oct 30 12:44:50.154916 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.154901 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/877cfa0b-b1c6-4455-b232-b1ca558b5084-nginx-conf\") pod \"networking-console-plugin-86c478bb88-5bmsj\" (UID: \"877cfa0b-b1c6-4455-b232-b1ca558b5084\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5bmsj" Oct 30 12:44:50.155014 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.154930 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cb39685-b929-48bd-afd2-228862e62668-metrics-tls\") pod \"dns-default-w52p8\" (UID: \"7cb39685-b929-48bd-afd2-228862e62668\") " pod="openshift-dns/dns-default-w52p8" Oct 30 12:44:50.155014 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.154963 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-tls\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:44:50.155014 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.154995 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-stats-auth\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:44:50.155120 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.155021 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-image-registry-private-configuration\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:44:50.155120 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.155047 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwpj2\" (UniqueName: \"kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-kube-api-access-jwpj2\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:44:50.155120 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.155091 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-sztw6\" (UID: \"73b3e3a7-93fa-49c2-a8f1-4188a22b99a5\") " pod="openshift-insights/insights-operator-7c4986d449-sztw6" Oct 30 12:44:50.157393 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.157377 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e4bf0a41-a4bd-47ad-93b1-088500499acb-serving-cert\") pod \"service-ca-operator-544d749c8c-kzhq7\" (UID: \"e4bf0a41-a4bd-47ad-93b1-088500499acb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzhq7" Oct 30 12:44:50.177370 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.177319 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbwtv\" (UniqueName: \"kubernetes.io/projected/9c377915-c344-4791-baf6-4b286b9c019b-kube-api-access-mbwtv\") pod \"kas-connection-checker-685895c7c9-6xnwh\" (UID: \"9c377915-c344-4791-baf6-4b286b9c019b\") " pod="kube-system/kas-connection-checker-685895c7c9-6xnwh" Oct 30 12:44:50.177454 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.177388 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcpmg\" (UniqueName: \"kubernetes.io/projected/e4bf0a41-a4bd-47ad-93b1-088500499acb-kube-api-access-lcpmg\") pod \"service-ca-operator-544d749c8c-kzhq7\" (UID: \"e4bf0a41-a4bd-47ad-93b1-088500499acb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzhq7" Oct 30 12:44:50.177454 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.177416 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf8zf\" (UniqueName: \"kubernetes.io/projected/a3928331-ce67-4c37-9c5d-63b0da642a0f-kube-api-access-vf8zf\") pod \"kas-connection-checker-685895c7c9-rnflh\" (UID: \"a3928331-ce67-4c37-9c5d-63b0da642a0f\") " pod="kube-system/kas-connection-checker-685895c7c9-rnflh" Oct 30 12:44:50.177454 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.177388 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnz9b\" (UniqueName: \"kubernetes.io/projected/ae9fe0c2-0d00-4891-8549-bc99f5373da9-kube-api-access-pnz9b\") pod \"kas-connection-checker-685895c7c9-cv78r\" (UID: \"ae9fe0c2-0d00-4891-8549-bc99f5373da9\") " pod="kube-system/kas-connection-checker-685895c7c9-cv78r" Oct 30 12:44:50.256024 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.255984 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-metrics-certs\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:44:50.256182 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.256036 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpzms\" (UniqueName: \"kubernetes.io/projected/1d06e291-b9d3-48f7-8fc2-f615440631d9-kube-api-access-jpzms\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:44:50.256182 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.256074 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00ee6aae-7264-4c8d-a337-c7c9f1132e9c-trusted-ca\") pod \"console-operator-7cd56b8dbc-v4mks\" (UID: \"00ee6aae-7264-4c8d-a337-c7c9f1132e9c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" Oct 30 12:44:50.256182 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.256100 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-trusted-ca\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:44:50.256182 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:50.256131 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 12:44:50.256182 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.256144 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-serving-cert\") pod \"insights-operator-7c4986d449-sztw6\" (UID: \"73b3e3a7-93fa-49c2-a8f1-4188a22b99a5\") " pod="openshift-insights/insights-operator-7c4986d449-sztw6" Oct 30 12:44:50.256182 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.256169 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cb39685-b929-48bd-afd2-228862e62668-metrics-tls\") pod \"dns-default-w52p8\" (UID: \"7cb39685-b929-48bd-afd2-228862e62668\") " pod="openshift-dns/dns-default-w52p8" Oct 30 12:44:50.256542 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:50.256197 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-metrics-certs podName:1d06e291-b9d3-48f7-8fc2-f615440631d9 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:50.756177919 +0000 UTC m=+30.661453433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-metrics-certs") pod "router-default-5db587c4bc-dttsw" (UID: "1d06e291-b9d3-48f7-8fc2-f615440631d9") : secret "router-metrics-certs-default" not found Oct 30 12:44:50.256542 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.256231 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-tls\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:44:50.256542 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.256259 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-stats-auth\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:44:50.256542 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:50.256287 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 12:44:50.256542 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.256303 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-image-registry-private-configuration\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:44:50.256542 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.256329 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwpj2\" (UniqueName: \"kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-kube-api-access-jwpj2\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:44:50.256542 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:50.256361 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cb39685-b929-48bd-afd2-228862e62668-metrics-tls podName:7cb39685-b929-48bd-afd2-228862e62668 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:50.75634508 +0000 UTC m=+30.661620599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cb39685-b929-48bd-afd2-228862e62668-metrics-tls") pod "dns-default-w52p8" (UID: "7cb39685-b929-48bd-afd2-228862e62668") : secret "dns-default-metrics-tls" not found Oct 30 12:44:50.256542 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:50.256365 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 12:44:50.256542 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:50.256380 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-596f5cc4c5-sgvjw: secret "image-registry-tls" not found Oct 30 12:44:50.256542 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.256383 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-sztw6\" (UID: \"73b3e3a7-93fa-49c2-a8f1-4188a22b99a5\") " pod="openshift-insights/insights-operator-7c4986d449-sztw6" Oct 30 12:44:50.256542 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.256414 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfff9\" (UniqueName: \"kubernetes.io/projected/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-kube-api-access-wfff9\") pod \"insights-operator-7c4986d449-sztw6\" (UID: \"73b3e3a7-93fa-49c2-a8f1-4188a22b99a5\") " pod="openshift-insights/insights-operator-7c4986d449-sztw6" Oct 30 12:44:50.256542 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.256441 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkdqk\" (UniqueName: \"kubernetes.io/projected/4df70e16-5acb-494e-ab67-855c809e3411-kube-api-access-hkdqk\") pod \"ingress-canary-dnpgs\" (UID: \"4df70e16-5acb-494e-ab67-855c809e3411\") " pod="openshift-ingress-canary/ingress-canary-dnpgs" Oct 30 12:44:50.256542 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:50.256450 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-tls podName:6000e43a-e175-46fe-b5f7-a417fc1ba5bb nodeName:}" failed. No retries permitted until 2026-10-30 12:44:50.756436793 +0000 UTC m=+30.661712309 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-tls") pod "image-registry-596f5cc4c5-sgvjw" (UID: "6000e43a-e175-46fe-b5f7-a417fc1ba5bb") : secret "image-registry-tls" not found Oct 30 12:44:50.256542 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.256481 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-tls\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:44:50.256542 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.256515 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-default-certificate\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:44:50.256542 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.256541 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-installation-pull-secrets\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:44:50.257776 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.256570 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/71e82393-029b-4388-86df-8bd068720867-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pptmg\" (UID: \"71e82393-029b-4388-86df-8bd068720867\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pptmg" Oct 30 12:44:50.257776 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.256594 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cb39685-b929-48bd-afd2-228862e62668-tmp-dir\") pod \"dns-default-w52p8\" (UID: \"7cb39685-b929-48bd-afd2-228862e62668\") " pod="openshift-dns/dns-default-w52p8" Oct 30 12:44:50.257776 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.256620 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbnzh\" (UniqueName: \"kubernetes.io/projected/71e82393-029b-4388-86df-8bd068720867-kube-api-access-kbnzh\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pptmg\" (UID: \"71e82393-029b-4388-86df-8bd068720867\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pptmg" Oct 30 12:44:50.257776 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.256641 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d06e291-b9d3-48f7-8fc2-f615440631d9-service-ca-bundle\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:44:50.257776 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.256675 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g248\" (UniqueName: \"kubernetes.io/projected/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-kube-api-access-4g248\") pod \"cluster-samples-operator-6ddb6c8b94-s7c7s\" (UID: \"4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-s7c7s" Oct 30 12:44:50.257776 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:50.256702 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 12:44:50.257776 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:50.256719 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bff985b5d-qwb5l: secret "image-registry-tls" not found Oct 30 12:44:50.257776 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:50.256765 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-tls podName:d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:50.756751623 +0000 UTC m=+30.662027143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-tls") pod "image-registry-7bff985b5d-qwb5l" (UID: "d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5") : secret "image-registry-tls" not found Oct 30 12:44:50.257776 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.256699 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpcwv\" (UniqueName: \"kubernetes.io/projected/8107811a-ec7c-497a-83fc-372249074496-kube-api-access-jpcwv\") pod \"network-check-source-6f78d59f99-pkpds\" (UID: \"8107811a-ec7c-497a-83fc-372249074496\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-pkpds" Oct 30 12:44:50.257776 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.257124 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00ee6aae-7264-4c8d-a337-c7c9f1132e9c-trusted-ca\") pod \"console-operator-7cd56b8dbc-v4mks\" (UID: \"00ee6aae-7264-4c8d-a337-c7c9f1132e9c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" Oct 30 12:44:50.257776 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.257140 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/00ee6aae-7264-4c8d-a337-c7c9f1132e9c-config\") pod \"console-operator-7cd56b8dbc-v4mks\" (UID: \"00ee6aae-7264-4c8d-a337-c7c9f1132e9c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" Oct 30 12:44:50.257776 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.257188 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wppl9\" (UniqueName: \"kubernetes.io/projected/64bdbc16-e845-45da-8f34-f0d788a75eac-kube-api-access-wppl9\") pod \"cluster-monitoring-operator-884576c57-vjdmx\" (UID: \"64bdbc16-e845-45da-8f34-f0d788a75eac\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjdmx" Oct 30 12:44:50.257776 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.257218 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-ca-trust-extracted\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:44:50.257776 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.257252 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-ca-trust-extracted\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:44:50.257776 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.257297 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4df70e16-5acb-494e-ab67-855c809e3411-cert\") pod \"ingress-canary-dnpgs\" (UID: \"4df70e16-5acb-494e-ab67-855c809e3411\") " pod="openshift-ingress-canary/ingress-canary-dnpgs" Oct 30 12:44:50.257776 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.257329 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6xkw\" (UniqueName: \"kubernetes.io/projected/7cb39685-b929-48bd-afd2-228862e62668-kube-api-access-p6xkw\") pod \"dns-default-w52p8\" (UID: \"7cb39685-b929-48bd-afd2-228862e62668\") " pod="openshift-dns/dns-default-w52p8" Oct 30 12:44:50.258593 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.257355 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-bound-sa-token\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:44:50.258593 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.257379 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7cb39685-b929-48bd-afd2-228862e62668-config-volume\") pod \"dns-default-w52p8\" (UID: \"7cb39685-b929-48bd-afd2-228862e62668\") " pod="openshift-dns/dns-default-w52p8" Oct 30 12:44:50.258593 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.257393 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-trusted-ca\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:44:50.258593 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.257403 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-bound-sa-token\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:44:50.258593 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.257482 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8wc5\" (UniqueName: \"kubernetes.io/projected/93e1032c-06a8-4418-99b7-540d1a4f1232-kube-api-access-p8wc5\") pod \"volume-data-source-validator-58c9f5f46d-c8pgj\" (UID: \"93e1032c-06a8-4418-99b7-540d1a4f1232\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-c8pgj" Oct 30 12:44:50.258593 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.257398 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-sztw6\" (UID: \"73b3e3a7-93fa-49c2-a8f1-4188a22b99a5\") " pod="openshift-insights/insights-operator-7c4986d449-sztw6" Oct 30 12:44:50.258593 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.257512 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-tmp\") pod \"insights-operator-7c4986d449-sztw6\" (UID: \"73b3e3a7-93fa-49c2-a8f1-4188a22b99a5\") " pod="openshift-insights/insights-operator-7c4986d449-sztw6" Oct 30 12:44:50.258593 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.257543 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-certificates\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:44:50.258593 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.257577 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/00ee6aae-7264-4c8d-a337-c7c9f1132e9c-serving-cert\") pod \"console-operator-7cd56b8dbc-v4mks\" (UID: \"00ee6aae-7264-4c8d-a337-c7c9f1132e9c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" Oct 30 12:44:50.258593 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.257615 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/64bdbc16-e845-45da-8f34-f0d788a75eac-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-vjdmx\" (UID: \"64bdbc16-e845-45da-8f34-f0d788a75eac\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjdmx" Oct 30 12:44:50.258593 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.257660 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/71e82393-029b-4388-86df-8bd068720867-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pptmg\" (UID: \"71e82393-029b-4388-86df-8bd068720867\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pptmg" Oct 30 12:44:50.258593 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.257685 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-s7c7s\" (UID: \"4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-s7c7s" Oct 30 12:44:50.258593 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.257714 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-trusted-ca\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:44:50.258593 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.257739 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-s7c7s\" (UID: \"4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-s7c7s" Oct 30 12:44:50.258593 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.257765 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-snapshots\") pod \"insights-operator-7c4986d449-sztw6\" (UID: \"73b3e3a7-93fa-49c2-a8f1-4188a22b99a5\") " pod="openshift-insights/insights-operator-7c4986d449-sztw6" Oct 30 12:44:50.258593 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.257770 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/00ee6aae-7264-4c8d-a337-c7c9f1132e9c-config\") pod \"console-operator-7cd56b8dbc-v4mks\" (UID: \"00ee6aae-7264-4c8d-a337-c7c9f1132e9c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" Oct 30 12:44:50.259408 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.258002 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-certificates\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:44:50.259408 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.258056 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-installation-pull-secrets\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:44:50.259408 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.258141 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7cb39685-b929-48bd-afd2-228862e62668-config-volume\") pod \"dns-default-w52p8\" (UID: \"7cb39685-b929-48bd-afd2-228862e62668\") " pod="openshift-dns/dns-default-w52p8" Oct 30 12:44:50.259408 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.258526 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-snapshots\") pod \"insights-operator-7c4986d449-sztw6\" (UID: \"73b3e3a7-93fa-49c2-a8f1-4188a22b99a5\") " pod="openshift-insights/insights-operator-7c4986d449-sztw6" Oct 30 12:44:50.259408 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.258648 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-image-registry-private-configuration\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:44:50.259408 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.258685 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zks6k\" (UniqueName: \"kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-kube-api-access-zks6k\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:44:50.259408 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.258721 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtcpz\" (UniqueName: \"kubernetes.io/projected/00ee6aae-7264-4c8d-a337-c7c9f1132e9c-kube-api-access-wtcpz\") pod \"console-operator-7cd56b8dbc-v4mks\" (UID: \"00ee6aae-7264-4c8d-a337-c7c9f1132e9c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" Oct 30 12:44:50.259408 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.258756 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-service-ca-bundle\") pod \"insights-operator-7c4986d449-sztw6\" (UID: \"73b3e3a7-93fa-49c2-a8f1-4188a22b99a5\") " pod="openshift-insights/insights-operator-7c4986d449-sztw6" Oct 30 12:44:50.259408 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.258790 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/64bdbc16-e845-45da-8f34-f0d788a75eac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vjdmx\" (UID: \"64bdbc16-e845-45da-8f34-f0d788a75eac\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjdmx" Oct 30 12:44:50.259408 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:50.258915 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 12:44:50.259408 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:50.258971 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64bdbc16-e845-45da-8f34-f0d788a75eac-cluster-monitoring-operator-tls podName:64bdbc16-e845-45da-8f34-f0d788a75eac nodeName:}" failed. No retries permitted until 2026-10-30 12:44:50.75895698 +0000 UTC m=+30.664232494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/64bdbc16-e845-45da-8f34-f0d788a75eac-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vjdmx" (UID: "64bdbc16-e845-45da-8f34-f0d788a75eac") : secret "cluster-monitoring-operator-tls" not found Oct 30 12:44:50.259408 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.259006 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-ca-trust-extracted\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:44:50.259408 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.259248 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-tmp\") pod \"insights-operator-7c4986d449-sztw6\" (UID: \"73b3e3a7-93fa-49c2-a8f1-4188a22b99a5\") " pod="openshift-insights/insights-operator-7c4986d449-sztw6" Oct 30 12:44:50.260027 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.259878 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-service-ca-bundle\") pod \"insights-operator-7c4986d449-sztw6\" (UID: \"73b3e3a7-93fa-49c2-a8f1-4188a22b99a5\") " pod="openshift-insights/insights-operator-7c4986d449-sztw6" Oct 30 12:44:50.260027 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.259942 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/64bdbc16-e845-45da-8f34-f0d788a75eac-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-vjdmx\" (UID: \"64bdbc16-e845-45da-8f34-f0d788a75eac\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjdmx" Oct 30 12:44:50.261679 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.260407 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-certificates\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:44:50.261679 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:50.260498 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 12:44:50.261679 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:50.260546 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-tls podName:4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c nodeName:}" failed. No retries permitted until 2026-10-30 12:44:50.760528639 +0000 UTC m=+30.665804139 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-s7c7s" (UID: "4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c") : secret "samples-operator-tls" not found Oct 30 12:44:50.261679 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.260806 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-ca-trust-extracted\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:44:50.261679 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.260972 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/71e82393-029b-4388-86df-8bd068720867-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pptmg\" (UID: \"71e82393-029b-4388-86df-8bd068720867\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pptmg" Oct 30 12:44:50.261679 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.261006 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cb39685-b929-48bd-afd2-228862e62668-tmp-dir\") pod \"dns-default-w52p8\" (UID: \"7cb39685-b929-48bd-afd2-228862e62668\") " pod="openshift-dns/dns-default-w52p8" Oct 30 12:44:50.261679 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:50.261112 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1d06e291-b9d3-48f7-8fc2-f615440631d9-service-ca-bundle podName:1d06e291-b9d3-48f7-8fc2-f615440631d9 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:50.761097793 +0000 UTC m=+30.666373314 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1d06e291-b9d3-48f7-8fc2-f615440631d9-service-ca-bundle") pod "router-default-5db587c4bc-dttsw" (UID: "1d06e291-b9d3-48f7-8fc2-f615440631d9") : configmap references non-existent config key: service-ca.crt Oct 30 12:44:50.261679 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.261244 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-trusted-ca\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:44:50.261679 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.261404 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-installation-pull-secrets\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:44:50.262224 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.261751 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-s7c7s\" (UID: \"4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-s7c7s" Oct 30 12:44:50.262224 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.261788 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-certificates\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:44:50.262462 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.262401 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4df70e16-5acb-494e-ab67-855c809e3411-cert\") pod \"ingress-canary-dnpgs\" (UID: \"4df70e16-5acb-494e-ab67-855c809e3411\") " pod="openshift-ingress-canary/ingress-canary-dnpgs" Oct 30 12:44:50.262917 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.262894 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-serving-cert\") pod \"insights-operator-7c4986d449-sztw6\" (UID: \"73b3e3a7-93fa-49c2-a8f1-4188a22b99a5\") " pod="openshift-insights/insights-operator-7c4986d449-sztw6" Oct 30 12:44:50.263025 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.263003 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-stats-auth\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:44:50.263206 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.263184 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-image-registry-private-configuration\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:44:50.263886 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.263858 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-default-certificate\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:44:50.263979 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.263951 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/71e82393-029b-4388-86df-8bd068720867-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pptmg\" (UID: \"71e82393-029b-4388-86df-8bd068720867\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pptmg" Oct 30 12:44:50.264153 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.264132 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/00ee6aae-7264-4c8d-a337-c7c9f1132e9c-serving-cert\") pod \"console-operator-7cd56b8dbc-v4mks\" (UID: \"00ee6aae-7264-4c8d-a337-c7c9f1132e9c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" Oct 30 12:44:50.264432 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.264409 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-image-registry-private-configuration\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:44:50.264881 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.264865 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-installation-pull-secrets\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:44:50.282347 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.282325 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkdqk\" (UniqueName: \"kubernetes.io/projected/4df70e16-5acb-494e-ab67-855c809e3411-kube-api-access-hkdqk\") pod \"ingress-canary-dnpgs\" (UID: \"4df70e16-5acb-494e-ab67-855c809e3411\") " pod="openshift-ingress-canary/ingress-canary-dnpgs" Oct 30 12:44:50.282477 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.282407 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpcwv\" (UniqueName: \"kubernetes.io/projected/8107811a-ec7c-497a-83fc-372249074496-kube-api-access-jpcwv\") pod \"network-check-source-6f78d59f99-pkpds\" (UID: \"8107811a-ec7c-497a-83fc-372249074496\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-pkpds" Oct 30 12:44:50.282836 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.282808 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpzms\" (UniqueName: \"kubernetes.io/projected/1d06e291-b9d3-48f7-8fc2-f615440631d9-kube-api-access-jpzms\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:44:50.282954 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.282931 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwpj2\" (UniqueName: \"kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-kube-api-access-jwpj2\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:44:50.284041 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.284017 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-bound-sa-token\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:44:50.284146 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.284099 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfff9\" (UniqueName: \"kubernetes.io/projected/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-kube-api-access-wfff9\") pod \"insights-operator-7c4986d449-sztw6\" (UID: \"73b3e3a7-93fa-49c2-a8f1-4188a22b99a5\") " pod="openshift-insights/insights-operator-7c4986d449-sztw6" Oct 30 12:44:50.284799 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.284770 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtcpz\" (UniqueName: \"kubernetes.io/projected/00ee6aae-7264-4c8d-a337-c7c9f1132e9c-kube-api-access-wtcpz\") pod \"console-operator-7cd56b8dbc-v4mks\" (UID: \"00ee6aae-7264-4c8d-a337-c7c9f1132e9c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" Oct 30 12:44:50.285606 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.285550 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbnzh\" (UniqueName: \"kubernetes.io/projected/71e82393-029b-4388-86df-8bd068720867-kube-api-access-kbnzh\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pptmg\" (UID: \"71e82393-029b-4388-86df-8bd068720867\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pptmg" Oct 30 12:44:50.285708 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.285632 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6xkw\" (UniqueName: \"kubernetes.io/projected/7cb39685-b929-48bd-afd2-228862e62668-kube-api-access-p6xkw\") pod \"dns-default-w52p8\" (UID: \"7cb39685-b929-48bd-afd2-228862e62668\") " pod="openshift-dns/dns-default-w52p8" Oct 30 12:44:50.286175 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.286151 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8wc5\" (UniqueName: \"kubernetes.io/projected/93e1032c-06a8-4418-99b7-540d1a4f1232-kube-api-access-p8wc5\") pod \"volume-data-source-validator-58c9f5f46d-c8pgj\" (UID: \"93e1032c-06a8-4418-99b7-540d1a4f1232\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-c8pgj" Oct 30 12:44:50.287387 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.287364 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zks6k\" (UniqueName: \"kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-kube-api-access-zks6k\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:44:50.287725 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.287705 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-rnflh" Oct 30 12:44:50.287809 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.287747 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-bound-sa-token\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:44:50.288389 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.288360 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wppl9\" (UniqueName: \"kubernetes.io/projected/64bdbc16-e845-45da-8f34-f0d788a75eac-kube-api-access-wppl9\") pod \"cluster-monitoring-operator-884576c57-vjdmx\" (UID: \"64bdbc16-e845-45da-8f34-f0d788a75eac\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjdmx" Oct 30 12:44:50.289665 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.289645 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g248\" (UniqueName: \"kubernetes.io/projected/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-kube-api-access-4g248\") pod \"cluster-samples-operator-6ddb6c8b94-s7c7s\" (UID: \"4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-s7c7s" Oct 30 12:44:50.296432 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.296123 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-cv78r" Oct 30 12:44:50.307481 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.307449 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6xnwh" Oct 30 12:44:50.319923 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.319892 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzhq7" Oct 30 12:44:50.332061 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.331466 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pptmg" Oct 30 12:44:50.354378 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.354345 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-sztw6" Oct 30 12:44:50.376182 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.375842 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-pkpds" Oct 30 12:44:50.388731 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.388544 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-c8pgj" Oct 30 12:44:50.404924 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.404842 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" Oct 30 12:44:50.410365 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.409483 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dnpgs" Oct 30 12:44:50.620709 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.620355 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cv78r"] Oct 30 12:44:50.625985 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:50.625952 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae9fe0c2_0d00_4891_8549_bc99f5373da9.slice/crio-532538a4de233945df82f5a57d2553398c6ca40852be60528ea13b458fb46270 WatchSource:0}: Error finding container 532538a4de233945df82f5a57d2553398c6ca40852be60528ea13b458fb46270: Status 404 returned error can't find the container with id 532538a4de233945df82f5a57d2553398c6ca40852be60528ea13b458fb46270 Oct 30 12:44:50.628247 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.628226 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6xnwh"] Oct 30 12:44:50.632138 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.632117 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-rnflh"] Oct 30 12:44:50.632461 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:50.632433 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c377915_c344_4791_baf6_4b286b9c019b.slice/crio-a7780b600c563f8797c365efb4bce1595f38972c8931e6286bbe49c4596ba076 WatchSource:0}: Error finding container a7780b600c563f8797c365efb4bce1595f38972c8931e6286bbe49c4596ba076: Status 404 returned error can't find the container with id a7780b600c563f8797c365efb4bce1595f38972c8931e6286bbe49c4596ba076 Oct 30 12:44:50.635622 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:50.635601 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3928331_ce67_4c37_9c5d_63b0da642a0f.slice/crio-0653b6460c0da538c8b59897aba69293f877ed4a6073a6228e50cc5d07cb7fd0 WatchSource:0}: Error finding container 0653b6460c0da538c8b59897aba69293f877ed4a6073a6228e50cc5d07cb7fd0: Status 404 returned error can't find the container with id 0653b6460c0da538c8b59897aba69293f877ed4a6073a6228e50cc5d07cb7fd0 Oct 30 12:44:50.663851 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.663809 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/877cfa0b-b1c6-4455-b232-b1ca558b5084-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5bmsj\" (UID: \"877cfa0b-b1c6-4455-b232-b1ca558b5084\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5bmsj" Oct 30 12:44:50.664122 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:50.663971 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 12:44:50.664244 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:50.664182 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/877cfa0b-b1c6-4455-b232-b1ca558b5084-networking-console-plugin-cert podName:877cfa0b-b1c6-4455-b232-b1ca558b5084 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:51.664161144 +0000 UTC m=+31.569436664 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/877cfa0b-b1c6-4455-b232-b1ca558b5084-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5bmsj" (UID: "877cfa0b-b1c6-4455-b232-b1ca558b5084") : secret "networking-console-plugin-cert" not found Oct 30 12:44:50.705835 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.705804 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-kzhq7"] Oct 30 12:44:50.706981 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.706897 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pptmg"] Oct 30 12:44:50.717079 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:50.717046 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71e82393_029b_4388_86df_8bd068720867.slice/crio-b216ecfdfe602b17806dca9305b70e41094ae20770a00460efa7dc75c45854bc WatchSource:0}: Error finding container b216ecfdfe602b17806dca9305b70e41094ae20770a00460efa7dc75c45854bc: Status 404 returned error can't find the container with id b216ecfdfe602b17806dca9305b70e41094ae20770a00460efa7dc75c45854bc Oct 30 12:44:50.717766 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:50.717734 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4bf0a41_a4bd_47ad_93b1_088500499acb.slice/crio-4b184290d081eb9feb51fc83eaab0b1960f37c98985e35ce272e3603ef62b343 WatchSource:0}: Error finding container 4b184290d081eb9feb51fc83eaab0b1960f37c98985e35ce272e3603ef62b343: Status 404 returned error can't find the container with id 4b184290d081eb9feb51fc83eaab0b1960f37c98985e35ce272e3603ef62b343 Oct 30 12:44:50.749089 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.747356 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-sztw6"] Oct 30 12:44:50.761753 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.761508 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-pkpds"] Oct 30 12:44:50.766809 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.766786 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/64bdbc16-e845-45da-8f34-f0d788a75eac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vjdmx\" (UID: \"64bdbc16-e845-45da-8f34-f0d788a75eac\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjdmx" Oct 30 12:44:50.767241 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.767227 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-metrics-certs\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:44:50.767515 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.767502 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cb39685-b929-48bd-afd2-228862e62668-metrics-tls\") pod \"dns-default-w52p8\" (UID: \"7cb39685-b929-48bd-afd2-228862e62668\") " pod="openshift-dns/dns-default-w52p8" Oct 30 12:44:50.767617 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.767607 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-tls\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:44:50.767793 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.767764 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-tls\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:44:50.767945 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.767925 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d06e291-b9d3-48f7-8fc2-f615440631d9-service-ca-bundle\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:44:50.768168 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.768149 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-s7c7s\" (UID: \"4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-s7c7s" Oct 30 12:44:50.768323 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:50.767172 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 12:44:50.768448 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:50.768428 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 12:44:50.768547 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:50.768533 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64bdbc16-e845-45da-8f34-f0d788a75eac-cluster-monitoring-operator-tls podName:64bdbc16-e845-45da-8f34-f0d788a75eac nodeName:}" failed. No retries permitted until 2026-10-30 12:44:51.768499997 +0000 UTC m=+31.673775512 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/64bdbc16-e845-45da-8f34-f0d788a75eac-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vjdmx" (UID: "64bdbc16-e845-45da-8f34-f0d788a75eac") : secret "cluster-monitoring-operator-tls" not found Oct 30 12:44:50.768631 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:50.768620 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-tls podName:4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c nodeName:}" failed. No retries permitted until 2026-10-30 12:44:51.768606601 +0000 UTC m=+31.673882101 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-s7c7s" (UID: "4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c") : secret "samples-operator-tls" not found Oct 30 12:44:50.768700 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:50.767841 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 12:44:50.768776 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:50.768762 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1d06e291-b9d3-48f7-8fc2-f615440631d9-service-ca-bundle podName:1d06e291-b9d3-48f7-8fc2-f615440631d9 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:51.768727707 +0000 UTC m=+31.674003222 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1d06e291-b9d3-48f7-8fc2-f615440631d9-service-ca-bundle") pod "router-default-5db587c4bc-dttsw" (UID: "1d06e291-b9d3-48f7-8fc2-f615440631d9") : configmap references non-existent config key: service-ca.crt Oct 30 12:44:50.768814 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:50.768791 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-metrics-certs podName:1d06e291-b9d3-48f7-8fc2-f615440631d9 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:51.768779471 +0000 UTC m=+31.674054977 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-metrics-certs") pod "router-default-5db587c4bc-dttsw" (UID: "1d06e291-b9d3-48f7-8fc2-f615440631d9") : secret "router-metrics-certs-default" not found Oct 30 12:44:50.768814 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:50.768210 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 12:44:50.768881 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:50.768823 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-596f5cc4c5-sgvjw: secret "image-registry-tls" not found Oct 30 12:44:50.768881 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:50.768261 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 12:44:50.768997 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:50.768981 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 12:44:50.769051 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:50.769044 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bff985b5d-qwb5l: secret "image-registry-tls" not found Oct 30 12:44:50.769139 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:50.768869 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-tls podName:6000e43a-e175-46fe-b5f7-a417fc1ba5bb nodeName:}" failed. No retries permitted until 2026-10-30 12:44:51.768850343 +0000 UTC m=+31.674125874 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-tls") pod "image-registry-596f5cc4c5-sgvjw" (UID: "6000e43a-e175-46fe-b5f7-a417fc1ba5bb") : secret "image-registry-tls" not found Oct 30 12:44:50.769216 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:50.769171 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cb39685-b929-48bd-afd2-228862e62668-metrics-tls podName:7cb39685-b929-48bd-afd2-228862e62668 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:51.769138973 +0000 UTC m=+31.674414474 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cb39685-b929-48bd-afd2-228862e62668-metrics-tls") pod "dns-default-w52p8" (UID: "7cb39685-b929-48bd-afd2-228862e62668") : secret "dns-default-metrics-tls" not found Oct 30 12:44:50.769216 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:50.769196 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-tls podName:d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:51.769184028 +0000 UTC m=+31.674459529 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-tls") pod "image-registry-7bff985b5d-qwb5l" (UID: "d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5") : secret "image-registry-tls" not found Oct 30 12:44:50.803573 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.803541 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-c8pgj"] Oct 30 12:44:50.807648 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:50.807624 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93e1032c_06a8_4418_99b7_540d1a4f1232.slice/crio-83241f9dbf1834814e3c065fb1d7d6086c72b4794461d53f9db893b020aa409e WatchSource:0}: Error finding container 83241f9dbf1834814e3c065fb1d7d6086c72b4794461d53f9db893b020aa409e: Status 404 returned error can't find the container with id 83241f9dbf1834814e3c065fb1d7d6086c72b4794461d53f9db893b020aa409e Oct 30 12:44:50.817192 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.817167 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6xnwh" event={"ID":"9c377915-c344-4791-baf6-4b286b9c019b","Type":"ContainerStarted","Data":"0e81408307982378d365fb102654f1028f24ce4ef378ae5fcf40236b32be972b"} Oct 30 12:44:50.817302 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.817218 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6xnwh" event={"ID":"9c377915-c344-4791-baf6-4b286b9c019b","Type":"ContainerStarted","Data":"a7780b600c563f8797c365efb4bce1595f38972c8931e6286bbe49c4596ba076"} Oct 30 12:44:50.818461 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.818418 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-v4mks"] Oct 30 12:44:50.819092 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.819072 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-cv78r" event={"ID":"ae9fe0c2-0d00-4891-8549-bc99f5373da9","Type":"ContainerStarted","Data":"7478b30cbcd0d2b57fe12036b7476a2220f782151147591a4de701e4de86a8af"} Oct 30 12:44:50.819154 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.819106 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-cv78r" event={"ID":"ae9fe0c2-0d00-4891-8549-bc99f5373da9","Type":"ContainerStarted","Data":"532538a4de233945df82f5a57d2553398c6ca40852be60528ea13b458fb46270"} Oct 30 12:44:50.820641 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.820622 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-rnflh" event={"ID":"a3928331-ce67-4c37-9c5d-63b0da642a0f","Type":"ContainerStarted","Data":"0238781cd3a25cf9140f902269eb83772d115f66d70e0438a09f771087e8dd8e"} Oct 30 12:44:50.820725 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.820658 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-rnflh" event={"ID":"a3928331-ce67-4c37-9c5d-63b0da642a0f","Type":"ContainerStarted","Data":"0653b6460c0da538c8b59897aba69293f877ed4a6073a6228e50cc5d07cb7fd0"} Oct 30 12:44:50.821755 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.821734 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-pkpds" event={"ID":"8107811a-ec7c-497a-83fc-372249074496","Type":"ContainerStarted","Data":"d450f953eb8dafb942c3939eddfdcca7a77b9b9d54e838fd3c3c588b8fc84f5d"} Oct 30 12:44:50.822086 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:50.822062 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00ee6aae_7264_4c8d_a337_c7c9f1132e9c.slice/crio-e4d8508e11cecca7ed2dbb5583af705e3979519f1ca6335b0bf42e686eddf99c WatchSource:0}: Error finding container e4d8508e11cecca7ed2dbb5583af705e3979519f1ca6335b0bf42e686eddf99c: Status 404 returned error can't find the container with id e4d8508e11cecca7ed2dbb5583af705e3979519f1ca6335b0bf42e686eddf99c Oct 30 12:44:50.822872 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.822803 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-sztw6" event={"ID":"73b3e3a7-93fa-49c2-a8f1-4188a22b99a5","Type":"ContainerStarted","Data":"1b6dccf7539753c2a0a42bbe3427f9581938d66159b1837a9871a2072ba8ad44"} Oct 30 12:44:50.826359 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.826314 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pptmg" event={"ID":"71e82393-029b-4388-86df-8bd068720867","Type":"ContainerStarted","Data":"b216ecfdfe602b17806dca9305b70e41094ae20770a00460efa7dc75c45854bc"} Oct 30 12:44:50.827772 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.827686 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-c8pgj" event={"ID":"93e1032c-06a8-4418-99b7-540d1a4f1232","Type":"ContainerStarted","Data":"83241f9dbf1834814e3c065fb1d7d6086c72b4794461d53f9db893b020aa409e"} Oct 30 12:44:50.828748 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.828728 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzhq7" event={"ID":"e4bf0a41-a4bd-47ad-93b1-088500499acb","Type":"ContainerStarted","Data":"4b184290d081eb9feb51fc83eaab0b1960f37c98985e35ce272e3603ef62b343"} Oct 30 12:44:50.846167 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.846147 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dnpgs"] Oct 30 12:44:50.850166 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:50.850145 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df70e16_5acb_494e_ab67_855c809e3411.slice/crio-c782a8370078036e2519d5d3f1e041769ae869568ed6a1a8f23b119f9a26daf3 WatchSource:0}: Error finding container c782a8370078036e2519d5d3f1e041769ae869568ed6a1a8f23b119f9a26daf3: Status 404 returned error can't find the container with id c782a8370078036e2519d5d3f1e041769ae869568ed6a1a8f23b119f9a26daf3 Oct 30 12:44:50.858168 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.858129 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-6xnwh" podStartSLOduration=420.858118418 podStartE2EDuration="7m0.858118418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:37:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:50.857453041 +0000 UTC m=+30.762728555" watchObservedRunningTime="2026-10-30 12:44:50.858118418 +0000 UTC m=+30.763393941" Oct 30 12:44:50.897666 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.897626 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-rnflh" podStartSLOduration=420.8976155 podStartE2EDuration="7m0.8976155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:37:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:50.897114365 +0000 UTC m=+30.802389885" watchObservedRunningTime="2026-10-30 12:44:50.8976155 +0000 UTC m=+30.802891023" Oct 30 12:44:50.937109 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:50.937065 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-cv78r" podStartSLOduration=420.93705496 podStartE2EDuration="7m0.93705496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:37:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:50.936936236 +0000 UTC m=+30.842211761" watchObservedRunningTime="2026-10-30 12:44:50.93705496 +0000 UTC m=+30.842330510" Oct 30 12:44:51.583228 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:51.583181 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:44:51.583792 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:51.583771 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:44:51.584311 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:51.584292 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljvqz" Oct 30 12:44:51.599555 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:51.599535 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 12:44:51.599845 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:51.599822 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5fhhc\"" Oct 30 12:44:51.599955 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:51.599944 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zngfb\"" Oct 30 12:44:51.600021 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:51.599997 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-w75sr\"" Oct 30 12:44:51.600081 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:51.600068 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 12:44:51.677409 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:51.676781 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/877cfa0b-b1c6-4455-b232-b1ca558b5084-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5bmsj\" (UID: \"877cfa0b-b1c6-4455-b232-b1ca558b5084\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5bmsj" Oct 30 12:44:51.677409 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:51.676979 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 12:44:51.677409 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:51.677044 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/877cfa0b-b1c6-4455-b232-b1ca558b5084-networking-console-plugin-cert podName:877cfa0b-b1c6-4455-b232-b1ca558b5084 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:53.677025827 +0000 UTC m=+33.582301335 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/877cfa0b-b1c6-4455-b232-b1ca558b5084-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5bmsj" (UID: "877cfa0b-b1c6-4455-b232-b1ca558b5084") : secret "networking-console-plugin-cert" not found Oct 30 12:44:51.779763 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:51.778021 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-s7c7s\" (UID: \"4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-s7c7s" Oct 30 12:44:51.779763 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:51.778115 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/64bdbc16-e845-45da-8f34-f0d788a75eac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vjdmx\" (UID: \"64bdbc16-e845-45da-8f34-f0d788a75eac\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjdmx" Oct 30 12:44:51.779763 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:51.778149 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-metrics-certs\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:44:51.779763 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:51.778194 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cb39685-b929-48bd-afd2-228862e62668-metrics-tls\") pod \"dns-default-w52p8\" (UID: \"7cb39685-b929-48bd-afd2-228862e62668\") " pod="openshift-dns/dns-default-w52p8" Oct 30 12:44:51.779763 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:51.778219 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-tls\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:44:51.779763 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:51.778290 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-tls\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:44:51.779763 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:51.778326 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d06e291-b9d3-48f7-8fc2-f615440631d9-service-ca-bundle\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:44:51.779763 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:51.778530 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1d06e291-b9d3-48f7-8fc2-f615440631d9-service-ca-bundle podName:1d06e291-b9d3-48f7-8fc2-f615440631d9 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:53.778509485 +0000 UTC m=+33.683785001 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1d06e291-b9d3-48f7-8fc2-f615440631d9-service-ca-bundle") pod "router-default-5db587c4bc-dttsw" (UID: "1d06e291-b9d3-48f7-8fc2-f615440631d9") : configmap references non-existent config key: service-ca.crt Oct 30 12:44:51.779763 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:51.778956 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 12:44:51.779763 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:51.779007 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-tls podName:4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c nodeName:}" failed. No retries permitted until 2026-10-30 12:44:53.778992776 +0000 UTC m=+33.684268279 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-s7c7s" (UID: "4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c") : secret "samples-operator-tls" not found Oct 30 12:44:51.779763 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:51.779072 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 12:44:51.779763 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:51.779106 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64bdbc16-e845-45da-8f34-f0d788a75eac-cluster-monitoring-operator-tls podName:64bdbc16-e845-45da-8f34-f0d788a75eac nodeName:}" failed. No retries permitted until 2026-10-30 12:44:53.779095567 +0000 UTC m=+33.684371069 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/64bdbc16-e845-45da-8f34-f0d788a75eac-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vjdmx" (UID: "64bdbc16-e845-45da-8f34-f0d788a75eac") : secret "cluster-monitoring-operator-tls" not found Oct 30 12:44:51.779763 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:51.779162 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 12:44:51.779763 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:51.779190 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-metrics-certs podName:1d06e291-b9d3-48f7-8fc2-f615440631d9 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:53.779180853 +0000 UTC m=+33.684456357 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-metrics-certs") pod "router-default-5db587c4bc-dttsw" (UID: "1d06e291-b9d3-48f7-8fc2-f615440631d9") : secret "router-metrics-certs-default" not found Oct 30 12:44:51.779763 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:51.779248 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 12:44:51.779763 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:51.779360 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 12:44:51.780650 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:51.779373 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bff985b5d-qwb5l: secret "image-registry-tls" not found Oct 30 12:44:51.780650 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:51.779395 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 12:44:51.780650 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:51.779407 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-tls podName:d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:53.77939663 +0000 UTC m=+33.684672137 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-tls") pod "image-registry-7bff985b5d-qwb5l" (UID: "d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5") : secret "image-registry-tls" not found Oct 30 12:44:51.780650 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:51.779413 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-596f5cc4c5-sgvjw: secret "image-registry-tls" not found Oct 30 12:44:51.780650 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:51.779456 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-tls podName:6000e43a-e175-46fe-b5f7-a417fc1ba5bb nodeName:}" failed. No retries permitted until 2026-10-30 12:44:53.779444283 +0000 UTC m=+33.684719812 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-tls") pod "image-registry-596f5cc4c5-sgvjw" (UID: "6000e43a-e175-46fe-b5f7-a417fc1ba5bb") : secret "image-registry-tls" not found Oct 30 12:44:51.780969 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:51.780945 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cb39685-b929-48bd-afd2-228862e62668-metrics-tls podName:7cb39685-b929-48bd-afd2-228862e62668 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:53.78092437 +0000 UTC m=+33.686199889 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cb39685-b929-48bd-afd2-228862e62668-metrics-tls") pod "dns-default-w52p8" (UID: "7cb39685-b929-48bd-afd2-228862e62668") : secret "dns-default-metrics-tls" not found Oct 30 12:44:51.834670 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:51.834587 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dnpgs" event={"ID":"4df70e16-5acb-494e-ab67-855c809e3411","Type":"ContainerStarted","Data":"c782a8370078036e2519d5d3f1e041769ae869568ed6a1a8f23b119f9a26daf3"} Oct 30 12:44:51.841872 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:51.841832 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" event={"ID":"00ee6aae-7264-4c8d-a337-c7c9f1132e9c","Type":"ContainerStarted","Data":"e4d8508e11cecca7ed2dbb5583af705e3979519f1ca6335b0bf42e686eddf99c"} Oct 30 12:44:53.700721 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:53.700669 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/877cfa0b-b1c6-4455-b232-b1ca558b5084-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5bmsj\" (UID: \"877cfa0b-b1c6-4455-b232-b1ca558b5084\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5bmsj" Oct 30 12:44:53.701451 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:53.700838 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 12:44:53.701451 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:53.700891 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/877cfa0b-b1c6-4455-b232-b1ca558b5084-networking-console-plugin-cert podName:877cfa0b-b1c6-4455-b232-b1ca558b5084 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:57.700876985 +0000 UTC m=+37.606152499 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/877cfa0b-b1c6-4455-b232-b1ca558b5084-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5bmsj" (UID: "877cfa0b-b1c6-4455-b232-b1ca558b5084") : secret "networking-console-plugin-cert" not found Oct 30 12:44:53.802654 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:53.801380 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/64bdbc16-e845-45da-8f34-f0d788a75eac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vjdmx\" (UID: \"64bdbc16-e845-45da-8f34-f0d788a75eac\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjdmx" Oct 30 12:44:53.802654 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:53.801446 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-metrics-certs\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:44:53.802654 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:53.801493 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cb39685-b929-48bd-afd2-228862e62668-metrics-tls\") pod \"dns-default-w52p8\" (UID: \"7cb39685-b929-48bd-afd2-228862e62668\") " pod="openshift-dns/dns-default-w52p8" Oct 30 12:44:53.802654 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:53.801519 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-tls\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:44:53.802654 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:53.801557 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-tls\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:44:53.802654 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:53.801586 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d06e291-b9d3-48f7-8fc2-f615440631d9-service-ca-bundle\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:44:53.802654 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:53.801649 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-s7c7s\" (UID: \"4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-s7c7s" Oct 30 12:44:53.802654 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:53.801756 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 12:44:53.802654 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:53.801803 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-tls podName:4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c nodeName:}" failed. No retries permitted until 2026-10-30 12:44:57.801791318 +0000 UTC m=+37.707066832 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-s7c7s" (UID: "4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c") : secret "samples-operator-tls" not found Oct 30 12:44:53.802654 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:53.802187 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 12:44:53.802654 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:53.802226 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64bdbc16-e845-45da-8f34-f0d788a75eac-cluster-monitoring-operator-tls podName:64bdbc16-e845-45da-8f34-f0d788a75eac nodeName:}" failed. No retries permitted until 2026-10-30 12:44:57.802216737 +0000 UTC m=+37.707492251 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/64bdbc16-e845-45da-8f34-f0d788a75eac-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vjdmx" (UID: "64bdbc16-e845-45da-8f34-f0d788a75eac") : secret "cluster-monitoring-operator-tls" not found Oct 30 12:44:53.802654 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:53.802281 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 12:44:53.802654 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:53.802316 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-metrics-certs podName:1d06e291-b9d3-48f7-8fc2-f615440631d9 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:57.802304914 +0000 UTC m=+37.707580437 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-metrics-certs") pod "router-default-5db587c4bc-dttsw" (UID: "1d06e291-b9d3-48f7-8fc2-f615440631d9") : secret "router-metrics-certs-default" not found Oct 30 12:44:53.802654 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:53.802371 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 12:44:53.802654 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:53.802403 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cb39685-b929-48bd-afd2-228862e62668-metrics-tls podName:7cb39685-b929-48bd-afd2-228862e62668 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:57.802393292 +0000 UTC m=+37.707668807 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cb39685-b929-48bd-afd2-228862e62668-metrics-tls") pod "dns-default-w52p8" (UID: "7cb39685-b929-48bd-afd2-228862e62668") : secret "dns-default-metrics-tls" not found Oct 30 12:44:53.805126 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:53.802532 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1d06e291-b9d3-48f7-8fc2-f615440631d9-service-ca-bundle podName:1d06e291-b9d3-48f7-8fc2-f615440631d9 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:57.80250171 +0000 UTC m=+37.707777228 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1d06e291-b9d3-48f7-8fc2-f615440631d9-service-ca-bundle") pod "router-default-5db587c4bc-dttsw" (UID: "1d06e291-b9d3-48f7-8fc2-f615440631d9") : configmap references non-existent config key: service-ca.crt Oct 30 12:44:53.805126 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:53.802606 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 12:44:53.805126 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:53.802619 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 12:44:53.805126 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:53.802633 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bff985b5d-qwb5l: secret "image-registry-tls" not found Oct 30 12:44:53.805126 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:53.802673 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-tls podName:d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:57.80266106 +0000 UTC m=+37.707936578 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-tls") pod "image-registry-7bff985b5d-qwb5l" (UID: "d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5") : secret "image-registry-tls" not found Oct 30 12:44:53.805126 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:53.802621 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-596f5cc4c5-sgvjw: secret "image-registry-tls" not found Oct 30 12:44:53.805126 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:53.802713 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-tls podName:6000e43a-e175-46fe-b5f7-a417fc1ba5bb nodeName:}" failed. No retries permitted until 2026-10-30 12:44:57.802703657 +0000 UTC m=+37.707979171 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-tls") pod "image-registry-596f5cc4c5-sgvjw" (UID: "6000e43a-e175-46fe-b5f7-a417fc1ba5bb") : secret "image-registry-tls" not found Oct 30 12:44:53.853560 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:53.853517 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-c8pgj" event={"ID":"93e1032c-06a8-4418-99b7-540d1a4f1232","Type":"ContainerStarted","Data":"53bc7efd65e3a36894305a5b1bd13c44047ca4a5963a1a9597ea58a02e5771a5"} Oct 30 12:44:53.898951 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:53.898886 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-c8pgj" podStartSLOduration=391.430153138 podStartE2EDuration="6m33.898868597s" totalImagesPullingTime="2.468715459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:38:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:53.897523487 +0000 UTC m=+33.802799010" watchObservedRunningTime="2026-10-30 12:44:53.898868597 +0000 UTC m=+33.804144122" Oct 30 12:44:54.409763 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:54.409677 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7664727a-bcd2-4169-b70b-a0ecba045321-metrics-certs\") pod \"network-metrics-daemon-j6b2w\" (UID: \"7664727a-bcd2-4169-b70b-a0ecba045321\") " pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:44:54.409942 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:54.409865 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 12:44:54.410001 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:54.409947 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7664727a-bcd2-4169-b70b-a0ecba045321-metrics-certs podName:7664727a-bcd2-4169-b70b-a0ecba045321 nodeName:}" failed. No retries permitted until 2026-10-30 12:45:26.409927301 +0000 UTC m=+66.315203038 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7664727a-bcd2-4169-b70b-a0ecba045321-metrics-certs") pod "network-metrics-daemon-j6b2w" (UID: "7664727a-bcd2-4169-b70b-a0ecba045321") : secret "metrics-daemon-secret" not found Oct 30 12:44:54.511968 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:54.510889 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4rqz\" (UniqueName: \"kubernetes.io/projected/eb4ec7a3-2fd7-44dd-98ca-72710972a050-kube-api-access-j4rqz\") pod \"network-check-target-ljvqz\" (UID: \"eb4ec7a3-2fd7-44dd-98ca-72710972a050\") " pod="openshift-network-diagnostics/network-check-target-ljvqz" Oct 30 12:44:54.511968 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:54.510995 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21ea018f-70e6-4ee3-952d-704089532abb-original-pull-secret\") pod \"global-pull-secret-syncer-nk9fb\" (UID: \"21ea018f-70e6-4ee3-952d-704089532abb\") " pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:44:54.513998 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:54.513969 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21ea018f-70e6-4ee3-952d-704089532abb-original-pull-secret\") pod \"global-pull-secret-syncer-nk9fb\" (UID: \"21ea018f-70e6-4ee3-952d-704089532abb\") " pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:44:54.514755 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:54.514698 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4rqz\" (UniqueName: \"kubernetes.io/projected/eb4ec7a3-2fd7-44dd-98ca-72710972a050-kube-api-access-j4rqz\") pod \"network-check-target-ljvqz\" (UID: \"eb4ec7a3-2fd7-44dd-98ca-72710972a050\") " pod="openshift-network-diagnostics/network-check-target-ljvqz" Oct 30 12:44:54.614632 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:54.614563 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:44:54.669121 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:54.669048 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljvqz" Oct 30 12:44:55.006456 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:55.006418 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ljvqz"] Oct 30 12:44:55.027168 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:55.027142 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nk9fb"] Oct 30 12:44:55.134753 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:55.134720 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ea018f_70e6_4ee3_952d_704089532abb.slice/crio-5b1db096190d30041623d13c301ebc223aa7dd6b6f4a2e6c5d8c83d8ddc2b25e WatchSource:0}: Error finding container 5b1db096190d30041623d13c301ebc223aa7dd6b6f4a2e6c5d8c83d8ddc2b25e: Status 404 returned error can't find the container with id 5b1db096190d30041623d13c301ebc223aa7dd6b6f4a2e6c5d8c83d8ddc2b25e Oct 30 12:44:55.860056 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:55.860020 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nk9fb" event={"ID":"21ea018f-70e6-4ee3-952d-704089532abb","Type":"ContainerStarted","Data":"5b1db096190d30041623d13c301ebc223aa7dd6b6f4a2e6c5d8c83d8ddc2b25e"} Oct 30 12:44:55.861195 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:55.861163 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ljvqz" event={"ID":"eb4ec7a3-2fd7-44dd-98ca-72710972a050","Type":"ContainerStarted","Data":"7a483e1e8ed42ec26ba81aa23d28315642ff2d3290e48d62cadb2abcf16dfbf6"} Oct 30 12:44:55.862796 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:55.862777 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/0.log" Oct 30 12:44:55.862890 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:55.862811 2574 generic.go:356] "Generic (PLEG): container finished" podID="00ee6aae-7264-4c8d-a337-c7c9f1132e9c" containerID="0ea77ead591210e80185187a372ba4d1e7506278e3df8366e5974d300310184d" exitCode=255 Oct 30 12:44:55.862890 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:55.862840 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" event={"ID":"00ee6aae-7264-4c8d-a337-c7c9f1132e9c","Type":"ContainerDied","Data":"0ea77ead591210e80185187a372ba4d1e7506278e3df8366e5974d300310184d"} Oct 30 12:44:55.863083 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:55.863068 2574 scope.go:122] "RemoveContainer" containerID="0ea77ead591210e80185187a372ba4d1e7506278e3df8366e5974d300310184d" Oct 30 12:44:56.883363 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:56.882535 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dnpgs" event={"ID":"4df70e16-5acb-494e-ab67-855c809e3411","Type":"ContainerStarted","Data":"ad77c590e151c83936b15b6c9bbe085d951febcb55821de27cd6acc4b8899351"} Oct 30 12:44:56.889697 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:56.889105 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-pkpds" event={"ID":"8107811a-ec7c-497a-83fc-372249074496","Type":"ContainerStarted","Data":"130ff1bd7e87008aa3a3f875bbb8131a8ab1f91902f4caed95d10f2bf9f224ea"} Oct 30 12:44:56.892294 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:56.891780 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-sztw6" event={"ID":"73b3e3a7-93fa-49c2-a8f1-4188a22b99a5","Type":"ContainerStarted","Data":"6c3473393491490b7e955cf33f49973c0223af60dfae4f6463a70d23709cb6ac"} Oct 30 12:44:56.894137 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:56.893707 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pptmg" event={"ID":"71e82393-029b-4388-86df-8bd068720867","Type":"ContainerStarted","Data":"6de6c456fb82b0d9abcd886b97ed82eacd6844d60d93bac83bc5ba18286d2f7b"} Oct 30 12:44:56.896206 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:56.895819 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzhq7" event={"ID":"e4bf0a41-a4bd-47ad-93b1-088500499acb","Type":"ContainerStarted","Data":"0c4d37ba87c40a37fe74f5eb9c86238d481fb22536037132f94335bb34ce6983"} Oct 30 12:44:56.899789 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:56.899701 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/1.log" Oct 30 12:44:56.900992 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:56.900118 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/0.log" Oct 30 12:44:56.900992 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:56.900148 2574 generic.go:356] "Generic (PLEG): container finished" podID="00ee6aae-7264-4c8d-a337-c7c9f1132e9c" containerID="c6678b84146d9fb4ce33ee83e569d21905f75a2151e695f43279698e8e6c13a8" exitCode=255 Oct 30 12:44:56.900992 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:56.900175 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" event={"ID":"00ee6aae-7264-4c8d-a337-c7c9f1132e9c","Type":"ContainerDied","Data":"c6678b84146d9fb4ce33ee83e569d21905f75a2151e695f43279698e8e6c13a8"} Oct 30 12:44:56.900992 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:56.900225 2574 scope.go:122] "RemoveContainer" containerID="0ea77ead591210e80185187a372ba4d1e7506278e3df8366e5974d300310184d" Oct 30 12:44:56.900992 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:56.900557 2574 scope.go:122] "RemoveContainer" containerID="c6678b84146d9fb4ce33ee83e569d21905f75a2151e695f43279698e8e6c13a8" Oct 30 12:44:56.900992 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:56.900712 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-v4mks_openshift-console-operator(00ee6aae-7264-4c8d-a337-c7c9f1132e9c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" podUID="00ee6aae-7264-4c8d-a337-c7c9f1132e9c" Oct 30 12:44:56.948521 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:56.947487 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dnpgs" podStartSLOduration=2.541513428 podStartE2EDuration="7.947470068s" totalImagesPullingTime="5.40595664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:56.924194485 +0000 UTC m=+36.829470009" watchObservedRunningTime="2026-10-30 12:44:56.947470068 +0000 UTC m=+36.852745595" Oct 30 12:44:56.977186 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:56.977122 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzhq7" podStartSLOduration=386.474873778 podStartE2EDuration="6m31.977102018s" totalImagesPullingTime="5.50222824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:38:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:56.976010153 +0000 UTC m=+36.881285676" watchObservedRunningTime="2026-10-30 12:44:56.977102018 +0000 UTC m=+36.882377542" Oct 30 12:44:57.070163 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.070045 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pptmg" podStartSLOduration=387.568347832 podStartE2EDuration="6m33.070022098s" totalImagesPullingTime="5.501674266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:57.021517147 +0000 UTC m=+36.926792671" watchObservedRunningTime="2026-10-30 12:44:57.070022098 +0000 UTC m=+36.975297622" Oct 30 12:44:57.124246 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.124150 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-sztw6" podStartSLOduration=392.669634316 podStartE2EDuration="6m38.124135705s" totalImagesPullingTime="5.454501389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:38:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:57.123614535 +0000 UTC m=+37.028890060" watchObservedRunningTime="2026-10-30 12:44:57.124135705 +0000 UTC m=+37.029411228" Oct 30 12:44:57.167562 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.167498 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-pkpds" podStartSLOduration=385.318598122 podStartE2EDuration="6m31.167477886s" totalImagesPullingTime="5.848879764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:57.166319577 +0000 UTC m=+37.071595097" watchObservedRunningTime="2026-10-30 12:44:57.167477886 +0000 UTC m=+37.072753409" Oct 30 12:44:57.254085 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.254051 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-b94k6"] Oct 30 12:44:57.286000 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.284914 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b94k6"] Oct 30 12:44:57.286000 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.285042 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:44:57.331160 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.331069 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 12:44:57.385180 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.385123 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 12:44:57.385180 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.385137 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g92fm\"" Oct 30 12:44:57.450776 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.450395 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40168acf-dcfa-4323-91eb-e710a2b82caa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:44:57.450776 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.450492 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/40168acf-dcfa-4323-91eb-e710a2b82caa-crio-socket\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:44:57.450776 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.450521 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40168acf-dcfa-4323-91eb-e710a2b82caa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:44:57.450776 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.450540 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/40168acf-dcfa-4323-91eb-e710a2b82caa-data-volume\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:44:57.450776 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.450642 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j94kk\" (UniqueName: \"kubernetes.io/projected/40168acf-dcfa-4323-91eb-e710a2b82caa-kube-api-access-j94kk\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:44:57.551679 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.551644 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j94kk\" (UniqueName: \"kubernetes.io/projected/40168acf-dcfa-4323-91eb-e710a2b82caa-kube-api-access-j94kk\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:44:57.551903 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.551700 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40168acf-dcfa-4323-91eb-e710a2b82caa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:44:57.551903 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.551778 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/40168acf-dcfa-4323-91eb-e710a2b82caa-crio-socket\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:44:57.551903 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.551803 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40168acf-dcfa-4323-91eb-e710a2b82caa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:44:57.551903 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.551822 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/40168acf-dcfa-4323-91eb-e710a2b82caa-data-volume\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:44:57.552121 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.552033 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/40168acf-dcfa-4323-91eb-e710a2b82caa-crio-socket\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:44:57.552121 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:57.552102 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 12:44:57.552216 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:57.552152 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/40168acf-dcfa-4323-91eb-e710a2b82caa-kube-rbac-proxy-cm podName:40168acf-dcfa-4323-91eb-e710a2b82caa nodeName:}" failed. No retries permitted until 2026-10-30 12:44:58.052132897 +0000 UTC m=+37.957408403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/40168acf-dcfa-4323-91eb-e710a2b82caa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b94k6" (UID: "40168acf-dcfa-4323-91eb-e710a2b82caa") : configmap "kube-rbac-proxy" not found Oct 30 12:44:57.552306 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.552254 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/40168acf-dcfa-4323-91eb-e710a2b82caa-data-volume\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:44:57.552504 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:57.552402 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 12:44:57.552504 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:57.552476 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40168acf-dcfa-4323-91eb-e710a2b82caa-insights-runtime-extractor-tls podName:40168acf-dcfa-4323-91eb-e710a2b82caa nodeName:}" failed. No retries permitted until 2026-10-30 12:44:58.05245797 +0000 UTC m=+37.957733475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40168acf-dcfa-4323-91eb-e710a2b82caa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b94k6" (UID: "40168acf-dcfa-4323-91eb-e710a2b82caa") : secret "insights-runtime-extractor-tls" not found Oct 30 12:44:57.582510 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.582439 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j94kk\" (UniqueName: \"kubernetes.io/projected/40168acf-dcfa-4323-91eb-e710a2b82caa-kube-api-access-j94kk\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:44:57.614210 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.614179 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4h8hg"] Oct 30 12:44:57.632726 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.632700 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4h8hg" Oct 30 12:44:57.659867 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.659793 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4h8hg"] Oct 30 12:44:57.689883 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.689844 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-k9mqg\"" Oct 30 12:44:57.709361 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.709278 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 12:44:57.709361 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.709314 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 12:44:57.753768 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.753733 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdkm8\" (UniqueName: \"kubernetes.io/projected/b32002db-049b-4456-91d7-66c33a85c040-kube-api-access-pdkm8\") pod \"migrator-66df5d496b-4h8hg\" (UID: \"b32002db-049b-4456-91d7-66c33a85c040\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4h8hg" Oct 30 12:44:57.753975 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.753912 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/877cfa0b-b1c6-4455-b232-b1ca558b5084-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5bmsj\" (UID: \"877cfa0b-b1c6-4455-b232-b1ca558b5084\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5bmsj" Oct 30 12:44:57.754075 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:57.754061 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 12:44:57.754126 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:57.754116 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/877cfa0b-b1c6-4455-b232-b1ca558b5084-networking-console-plugin-cert podName:877cfa0b-b1c6-4455-b232-b1ca558b5084 nodeName:}" failed. No retries permitted until 2026-10-30 12:45:05.75410072 +0000 UTC m=+45.659376221 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/877cfa0b-b1c6-4455-b232-b1ca558b5084-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5bmsj" (UID: "877cfa0b-b1c6-4455-b232-b1ca558b5084") : secret "networking-console-plugin-cert" not found Oct 30 12:44:57.854975 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.854892 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/64bdbc16-e845-45da-8f34-f0d788a75eac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vjdmx\" (UID: \"64bdbc16-e845-45da-8f34-f0d788a75eac\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjdmx" Oct 30 12:44:57.854975 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.854947 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-metrics-certs\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:44:57.854975 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.854972 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cb39685-b929-48bd-afd2-228862e62668-metrics-tls\") pod \"dns-default-w52p8\" (UID: \"7cb39685-b929-48bd-afd2-228862e62668\") " pod="openshift-dns/dns-default-w52p8" Oct 30 12:44:57.855240 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.854990 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-tls\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:44:57.855240 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.855012 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-tls\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:44:57.855240 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.855033 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d06e291-b9d3-48f7-8fc2-f615440631d9-service-ca-bundle\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:44:57.855240 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.855050 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdkm8\" (UniqueName: \"kubernetes.io/projected/b32002db-049b-4456-91d7-66c33a85c040-kube-api-access-pdkm8\") pod \"migrator-66df5d496b-4h8hg\" (UID: \"b32002db-049b-4456-91d7-66c33a85c040\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4h8hg" Oct 30 12:44:57.855240 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.855128 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-s7c7s\" (UID: \"4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-s7c7s" Oct 30 12:44:57.855536 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:57.855244 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 12:44:57.855536 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:57.855317 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-tls podName:4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c nodeName:}" failed. No retries permitted until 2026-10-30 12:45:05.855299057 +0000 UTC m=+45.760574560 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-s7c7s" (UID: "4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c") : secret "samples-operator-tls" not found Oct 30 12:44:57.855536 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:57.855385 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 12:44:57.855536 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:57.855429 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-metrics-certs podName:1d06e291-b9d3-48f7-8fc2-f615440631d9 nodeName:}" failed. No retries permitted until 2026-10-30 12:45:05.855416329 +0000 UTC m=+45.760691844 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-metrics-certs") pod "router-default-5db587c4bc-dttsw" (UID: "1d06e291-b9d3-48f7-8fc2-f615440631d9") : secret "router-metrics-certs-default" not found Oct 30 12:44:57.855536 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:57.855459 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1d06e291-b9d3-48f7-8fc2-f615440631d9-service-ca-bundle podName:1d06e291-b9d3-48f7-8fc2-f615440631d9 nodeName:}" failed. No retries permitted until 2026-10-30 12:45:05.855437337 +0000 UTC m=+45.760712846 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1d06e291-b9d3-48f7-8fc2-f615440631d9-service-ca-bundle") pod "router-default-5db587c4bc-dttsw" (UID: "1d06e291-b9d3-48f7-8fc2-f615440631d9") : configmap references non-existent config key: service-ca.crt Oct 30 12:44:57.855536 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:57.855482 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 12:44:57.855536 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:57.855513 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cb39685-b929-48bd-afd2-228862e62668-metrics-tls podName:7cb39685-b929-48bd-afd2-228862e62668 nodeName:}" failed. No retries permitted until 2026-10-30 12:45:05.855502431 +0000 UTC m=+45.760777966 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cb39685-b929-48bd-afd2-228862e62668-metrics-tls") pod "dns-default-w52p8" (UID: "7cb39685-b929-48bd-afd2-228862e62668") : secret "dns-default-metrics-tls" not found Oct 30 12:44:57.855536 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:57.855525 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 12:44:57.855536 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:57.855540 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-596f5cc4c5-sgvjw: secret "image-registry-tls" not found Oct 30 12:44:57.855951 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:57.855558 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 12:44:57.855951 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:57.855575 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-tls podName:6000e43a-e175-46fe-b5f7-a417fc1ba5bb nodeName:}" failed. No retries permitted until 2026-10-30 12:45:05.85556318 +0000 UTC m=+45.760838687 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-tls") pod "image-registry-596f5cc4c5-sgvjw" (UID: "6000e43a-e175-46fe-b5f7-a417fc1ba5bb") : secret "image-registry-tls" not found Oct 30 12:44:57.855951 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:57.855589 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64bdbc16-e845-45da-8f34-f0d788a75eac-cluster-monitoring-operator-tls podName:64bdbc16-e845-45da-8f34-f0d788a75eac nodeName:}" failed. No retries permitted until 2026-10-30 12:45:05.855580161 +0000 UTC m=+45.760855701 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/64bdbc16-e845-45da-8f34-f0d788a75eac-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vjdmx" (UID: "64bdbc16-e845-45da-8f34-f0d788a75eac") : secret "cluster-monitoring-operator-tls" not found Oct 30 12:44:57.855951 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:57.855633 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 12:44:57.855951 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:57.855642 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bff985b5d-qwb5l: secret "image-registry-tls" not found Oct 30 12:44:57.855951 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:57.855671 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-tls podName:d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5 nodeName:}" failed. No retries permitted until 2026-10-30 12:45:05.855661668 +0000 UTC m=+45.760937170 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-tls") pod "image-registry-7bff985b5d-qwb5l" (UID: "d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5") : secret "image-registry-tls" not found Oct 30 12:44:57.882300 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.882250 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdkm8\" (UniqueName: \"kubernetes.io/projected/b32002db-049b-4456-91d7-66c33a85c040-kube-api-access-pdkm8\") pod \"migrator-66df5d496b-4h8hg\" (UID: \"b32002db-049b-4456-91d7-66c33a85c040\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4h8hg" Oct 30 12:44:57.908578 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.908549 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/1.log" Oct 30 12:44:57.909641 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.909619 2574 scope.go:122] "RemoveContainer" containerID="c6678b84146d9fb4ce33ee83e569d21905f75a2151e695f43279698e8e6c13a8" Oct 30 12:44:57.909875 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:57.909840 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-v4mks_openshift-console-operator(00ee6aae-7264-4c8d-a337-c7c9f1132e9c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" podUID="00ee6aae-7264-4c8d-a337-c7c9f1132e9c" Oct 30 12:44:57.959921 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:57.959887 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4h8hg" Oct 30 12:44:58.058026 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:58.057989 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40168acf-dcfa-4323-91eb-e710a2b82caa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:44:58.058185 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:58.058150 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40168acf-dcfa-4323-91eb-e710a2b82caa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:44:58.058391 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:58.058366 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 12:44:58.058518 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:58.058442 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/40168acf-dcfa-4323-91eb-e710a2b82caa-kube-rbac-proxy-cm podName:40168acf-dcfa-4323-91eb-e710a2b82caa nodeName:}" failed. No retries permitted until 2026-10-30 12:44:59.058422725 +0000 UTC m=+38.963698243 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/40168acf-dcfa-4323-91eb-e710a2b82caa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b94k6" (UID: "40168acf-dcfa-4323-91eb-e710a2b82caa") : configmap "kube-rbac-proxy" not found Oct 30 12:44:58.058518 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:58.058442 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 12:44:58.058518 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:58.058479 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40168acf-dcfa-4323-91eb-e710a2b82caa-insights-runtime-extractor-tls podName:40168acf-dcfa-4323-91eb-e710a2b82caa nodeName:}" failed. No retries permitted until 2026-10-30 12:44:59.05846978 +0000 UTC m=+38.963745282 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40168acf-dcfa-4323-91eb-e710a2b82caa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b94k6" (UID: "40168acf-dcfa-4323-91eb-e710a2b82caa") : secret "insights-runtime-extractor-tls" not found Oct 30 12:44:58.275433 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:58.275402 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4h8hg"] Oct 30 12:44:58.279816 ip-10-0-142-194 kubenswrapper[2574]: W1030 12:44:58.279783 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb32002db_049b_4456_91d7_66c33a85c040.slice/crio-115ad062fe20e4014413c9535f123fd48eb68e3ede1918a04b12d1dde106cc8f WatchSource:0}: Error finding container 115ad062fe20e4014413c9535f123fd48eb68e3ede1918a04b12d1dde106cc8f: Status 404 returned error can't find the container with id 115ad062fe20e4014413c9535f123fd48eb68e3ede1918a04b12d1dde106cc8f Oct 30 12:44:58.913990 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:58.913881 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ljvqz" event={"ID":"eb4ec7a3-2fd7-44dd-98ca-72710972a050","Type":"ContainerStarted","Data":"94b35b020e260d4a78768e28d230d6a6d1b1eaa13cd0842aa0652ae47fe26bab"} Oct 30 12:44:58.913990 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:58.913975 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ljvqz" Oct 30 12:44:58.915695 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:58.915640 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4h8hg" event={"ID":"b32002db-049b-4456-91d7-66c33a85c040","Type":"ContainerStarted","Data":"115ad062fe20e4014413c9535f123fd48eb68e3ede1918a04b12d1dde106cc8f"} Oct 30 12:44:58.956081 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:58.956027 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ljvqz" podStartSLOduration=35.609807383 podStartE2EDuration="38.95600787s" totalImagesPullingTime="3.346200487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:44:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:58.954838861 +0000 UTC m=+38.860114386" watchObservedRunningTime="2026-10-30 12:44:58.95600787 +0000 UTC m=+38.861283393" Oct 30 12:44:59.067556 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:59.067521 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40168acf-dcfa-4323-91eb-e710a2b82caa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:44:59.067725 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:59.067634 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40168acf-dcfa-4323-91eb-e710a2b82caa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:44:59.067725 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:59.067654 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 12:44:59.067810 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:59.067729 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/40168acf-dcfa-4323-91eb-e710a2b82caa-kube-rbac-proxy-cm podName:40168acf-dcfa-4323-91eb-e710a2b82caa nodeName:}" failed. No retries permitted until 2026-10-30 12:45:01.0677134 +0000 UTC m=+40.972988928 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/40168acf-dcfa-4323-91eb-e710a2b82caa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b94k6" (UID: "40168acf-dcfa-4323-91eb-e710a2b82caa") : configmap "kube-rbac-proxy" not found Oct 30 12:44:59.067810 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:59.067759 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 12:44:59.067810 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:44:59.067801 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40168acf-dcfa-4323-91eb-e710a2b82caa-insights-runtime-extractor-tls podName:40168acf-dcfa-4323-91eb-e710a2b82caa nodeName:}" failed. No retries permitted until 2026-10-30 12:45:01.067789831 +0000 UTC m=+40.973065344 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40168acf-dcfa-4323-91eb-e710a2b82caa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b94k6" (UID: "40168acf-dcfa-4323-91eb-e710a2b82caa") : secret "insights-runtime-extractor-tls" not found Oct 30 12:44:59.921037 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:59.920997 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4h8hg" event={"ID":"b32002db-049b-4456-91d7-66c33a85c040","Type":"ContainerStarted","Data":"678389e5856247f4494eca120bf94fd0d34dcba04eef9738bd3f2121fefbcf06"} Oct 30 12:44:59.962694 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:44:59.962636 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4h8hg" podStartSLOduration=1.462677886 podStartE2EDuration="2.962619079s" totalImagesPullingTime="1.499941193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:44:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:59.962565368 +0000 UTC m=+39.867840893" watchObservedRunningTime="2026-10-30 12:44:59.962619079 +0000 UTC m=+39.867894604" Oct 30 12:45:00.405944 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:45:00.405913 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" Oct 30 12:45:00.406111 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:45:00.405976 2574 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" Oct 30 12:45:00.406365 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:45:00.406348 2574 scope.go:122] "RemoveContainer" containerID="c6678b84146d9fb4ce33ee83e569d21905f75a2151e695f43279698e8e6c13a8" Oct 30 12:45:00.406575 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:45:00.406549 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-v4mks_openshift-console-operator(00ee6aae-7264-4c8d-a337-c7c9f1132e9c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" podUID="00ee6aae-7264-4c8d-a337-c7c9f1132e9c" Oct 30 12:45:01.090344 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:45:01.090313 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40168acf-dcfa-4323-91eb-e710a2b82caa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:45:01.090621 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:45:01.090450 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 12:45:01.090621 ip-10-0-142-194 kubenswrapper[2574]: I1030 12:45:01.090471 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40168acf-dcfa-4323-91eb-e710a2b82caa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:45:01.090621 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:45:01.090565 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40168acf-dcfa-4323-91eb-e710a2b82caa-insights-runtime-extractor-tls podName:40168acf-dcfa-4323-91eb-e710a2b82caa nodeName:}" failed. No retries permitted until 2026-10-30 12:45:05.090543695 +0000 UTC m=+44.995819197 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40168acf-dcfa-4323-91eb-e710a2b82caa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b94k6" (UID: "40168acf-dcfa-4323-91eb-e710a2b82caa") : secret "insights-runtime-extractor-tls" not found Oct 30 12:45:01.090621 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:45:01.090609 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 12:45:01.090773 ip-10-0-142-194 kubenswrapper[2574]: E1030 12:45:01.090667 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/40168acf-dcfa-4323-91eb-e710a2b82caa-kube-rbac-proxy-cm podName:40168acf-dcfa-4323-91eb-e710a2b82caa nodeName:}" failed. No retries permitted until 2026-10-30 12:45:05.090648723 +0000 UTC m=+44.995924228 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/40168acf-dcfa-4323-91eb-e710a2b82caa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b94k6" (UID: "40168acf-dcfa-4323-91eb-e710a2b82caa") : configmap "kube-rbac-proxy" not found Oct 30 12:45:01.317304 ip-10-0-142-194 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 12:45:01.337653 ip-10-0-142-194 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 12:45:01.337816 ip-10-0-142-194 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 12:45:01.337876 ip-10-0-142-194 systemd[1]: kubelet.service: Consumed 3.012s CPU time. Oct 30 12:45:01.351901 ip-10-0-142-194 systemd[1]: Starting Kubernetes Kubelet... Oct 30 12:45:01.425465 ip-10-0-142-194 kubenswrapper[4931]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:45:01.425465 ip-10-0-142-194 kubenswrapper[4931]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 12:45:01.425465 ip-10-0-142-194 kubenswrapper[4931]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:45:01.425465 ip-10-0-142-194 kubenswrapper[4931]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:45:01.427819 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427738 4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:45:01.427819 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427815 4931 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:45:01.427819 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427821 4931 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:45:01.428029 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427827 4931 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:45:01.428029 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427832 4931 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:45:01.428029 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427836 4931 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:45:01.428029 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427841 4931 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:45:01.428029 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427845 4931 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:45:01.428029 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427849 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:45:01.428029 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427853 4931 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:45:01.428029 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427857 4931 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:45:01.428029 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427862 4931 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:45:01.428029 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427866 4931 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:45:01.428029 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427870 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:45:01.428029 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427874 4931 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:45:01.428029 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427878 4931 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:45:01.428029 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427882 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:45:01.428029 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427887 4931 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:45:01.428029 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427895 4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:45:01.428029 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427900 4931 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:45:01.428029 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427905 4931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:45:01.428029 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427908 4931 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:45:01.428029 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427912 4931 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:45:01.428920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427918 4931 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:45:01.428920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427924 4931 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:45:01.428920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427929 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:45:01.428920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427933 4931 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:45:01.428920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427937 4931 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:45:01.428920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427942 4931 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:45:01.428920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427946 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:45:01.428920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427950 4931 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:45:01.428920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427955 4931 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:45:01.428920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427960 4931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:45:01.428920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427964 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:45:01.428920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427969 4931 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:45:01.428920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427974 4931 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:45:01.428920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427979 4931 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:45:01.428920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.427995 4931 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:45:01.428920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428001 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:45:01.428920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428006 4931 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:45:01.428920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428010 4931 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:45:01.428920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428014 4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:45:01.429760 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428019 4931 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:45:01.429760 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428023 4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:45:01.429760 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428028 4931 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:45:01.429760 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428032 4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:45:01.429760 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428036 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:45:01.429760 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428040 4931 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:45:01.429760 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428045 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:45:01.429760 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428049 4931 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:45:01.429760 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428053 4931 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:45:01.429760 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428058 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:45:01.429760 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428062 4931 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:45:01.429760 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428066 4931 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:45:01.429760 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428071 4931 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:45:01.429760 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428075 4931 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:45:01.429760 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428080 4931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:45:01.429760 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428084 4931 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:45:01.429760 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428088 4931 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:45:01.429760 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428093 4931 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:45:01.429760 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428097 4931 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:45:01.429760 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428101 4931 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:45:01.430669 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428106 4931 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:45:01.430669 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428110 4931 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:45:01.430669 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428115 4931 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:45:01.430669 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428119 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:45:01.430669 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428123 4931 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:45:01.430669 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428127 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:45:01.430669 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428132 4931 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:45:01.430669 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428136 4931 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:45:01.430669 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428149 4931 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:45:01.430669 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428155 4931 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:45:01.430669 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428160 4931 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:45:01.430669 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428164 4931 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:45:01.430669 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428169 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:45:01.430669 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428174 4931 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:45:01.430669 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428178 4931 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:45:01.430669 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428182 4931 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:45:01.430669 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428186 4931 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:45:01.430669 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428191 4931 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:45:01.430669 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428195 4931 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:45:01.431512 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428200 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:45:01.431512 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428204 4931 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:45:01.431512 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428209 4931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:45:01.431512 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428213 4931 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:45:01.431512 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428218 4931 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:45:01.431512 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428222 4931 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:45:01.431512 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428227 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:45:01.431512 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428231 4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:45:01.431512 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428235 4931 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:45:01.431512 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428239 4931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:45:01.431512 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428244 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:45:01.431512 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428251 4931 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:45:01.431512 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428256 4931 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:45:01.431512 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428261 4931 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:45:01.431512 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428282 4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:45:01.431512 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428287 4931 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:45:01.431512 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428291 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:45:01.431512 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428295 4931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:45:01.431512 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428299 4931 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:45:01.432378 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428304 4931 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:45:01.432378 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428308 4931 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:45:01.432378 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428313 4931 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:45:01.432378 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428326 4931 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:45:01.432378 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428331 4931 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:45:01.432378 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.428335 4931 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:45:01.432378 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429705 4931 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:45:01.432378 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429716 4931 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:45:01.432378 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429720 4931 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:45:01.432378 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429725 4931 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:45:01.432378 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429729 4931 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:45:01.432378 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429734 4931 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:45:01.432378 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429738 4931 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:45:01.432378 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429744 4931 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:45:01.432378 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429750 4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:45:01.432378 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429755 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:45:01.432378 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429759 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:45:01.432378 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429763 4931 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:45:01.432378 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429768 4931 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:45:01.433224 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429772 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:45:01.433224 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429776 4931 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:45:01.433224 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429780 4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:45:01.433224 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429784 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:45:01.433224 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429789 4931 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:45:01.433224 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429793 4931 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:45:01.433224 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429798 4931 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:45:01.433224 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429804 4931 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:45:01.433224 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429810 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:45:01.433224 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429815 4931 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:45:01.433224 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429820 4931 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:45:01.433224 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429824 4931 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:45:01.433224 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429829 4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:45:01.433224 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429834 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:45:01.433224 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429839 4931 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:45:01.433224 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429843 4931 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:45:01.433224 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429848 4931 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:45:01.433224 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429862 4931 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:45:01.433224 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429866 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:45:01.433224 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429870 4931 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:45:01.434186 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429875 4931 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:45:01.434186 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429879 4931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:45:01.434186 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429883 4931 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:45:01.434186 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429888 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:45:01.434186 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429892 4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:45:01.434186 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429901 4931 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:45:01.434186 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429906 4931 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:45:01.434186 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429911 4931 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:45:01.434186 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429916 4931 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:45:01.434186 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429920 4931 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:45:01.434186 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429924 4931 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:45:01.434186 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429928 4931 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:45:01.434186 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429932 4931 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:45:01.434186 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429937 4931 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:45:01.434186 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429941 4931 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:45:01.434186 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429946 4931 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:45:01.434186 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429950 4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:45:01.434186 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429954 4931 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:45:01.434186 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429958 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:45:01.434801 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429963 4931 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:45:01.434801 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429967 4931 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:45:01.434801 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429971 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:45:01.434801 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429976 4931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:45:01.434801 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429980 4931 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:45:01.434801 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429985 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:45:01.434801 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429989 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:45:01.434801 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429993 4931 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:45:01.434801 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.429998 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:45:01.434801 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430002 4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:45:01.434801 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430006 4931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:45:01.434801 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430011 4931 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:45:01.434801 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430016 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:45:01.434801 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430020 4931 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:45:01.434801 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430024 4931 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:45:01.434801 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430028 4931 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:45:01.434801 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430032 4931 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:45:01.434801 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430037 4931 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:45:01.434801 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430043 4931 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:45:01.434801 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430047 4931 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:45:01.435318 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430051 4931 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:45:01.435318 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430056 4931 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:45:01.435318 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430060 4931 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:45:01.435318 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430065 4931 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:45:01.435318 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430069 4931 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:45:01.435318 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430074 4931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:45:01.435318 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430078 4931 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:45:01.435318 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430082 4931 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:45:01.435318 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430089 4931 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:45:01.435318 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430096 4931 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:45:01.435318 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430101 4931 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:45:01.435318 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430105 4931 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:45:01.435318 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430110 4931 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:45:01.435318 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430116 4931 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:45:01.435318 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430120 4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:45:01.435318 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430124 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:45:01.435318 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430129 4931 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:45:01.435318 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430133 4931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:45:01.435318 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430137 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:45:01.435789 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430141 4931 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:45:01.435789 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430145 4931 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:45:01.435789 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430150 4931 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:45:01.435789 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430154 4931 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:45:01.435789 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430158 4931 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:45:01.435789 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430163 4931 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:45:01.435789 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430167 4931 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:45:01.435789 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430172 4931 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:45:01.435789 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430176 4931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:45:01.435789 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430180 4931 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:45:01.435789 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430184 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:45:01.435789 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430190 4931 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:45:01.435789 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430195 4931 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:45:01.435789 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430200 4931 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:45:01.435789 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.430204 4931 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:45:01.435789 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431070 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:45:01.435789 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431080 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:45:01.435789 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431085 4931 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:45:01.435789 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431089 4931 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:45:01.436330 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431094 4931 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:45:01.436330 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431098 4931 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:45:01.436330 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431102 4931 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:45:01.436330 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431107 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:45:01.436330 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431111 4931 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:45:01.436330 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431116 4931 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:45:01.436330 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431121 4931 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:45:01.436330 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431125 4931 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:45:01.436330 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431130 4931 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:45:01.436330 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431135 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:45:01.436330 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431140 4931 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:45:01.436330 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431144 4931 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:45:01.436330 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431148 4931 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:45:01.436330 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431153 4931 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:45:01.436330 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431158 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:45:01.436330 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431162 4931 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:45:01.436330 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431167 4931 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:45:01.436330 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431171 4931 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:45:01.436330 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431184 4931 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:45:01.436330 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431188 4931 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:45:01.436918 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431193 4931 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:45:01.436918 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431197 4931 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:45:01.436918 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431201 4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:45:01.436918 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431205 4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:45:01.436918 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431212 4931 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:45:01.436918 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431217 4931 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:45:01.436918 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431222 4931 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:45:01.436918 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431226 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:45:01.436918 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431230 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:45:01.436918 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431234 4931 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:45:01.436918 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431239 4931 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:45:01.436918 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431243 4931 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:45:01.436918 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431248 4931 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:45:01.436918 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431252 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:45:01.436918 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431257 4931 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:45:01.436918 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431277 4931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:45:01.436918 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431282 4931 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:45:01.436918 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431287 4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:45:01.436918 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431291 4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:45:01.437688 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431295 4931 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:45:01.437688 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431300 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:45:01.437688 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431305 4931 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:45:01.437688 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431310 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:45:01.437688 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431314 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:45:01.437688 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431318 4931 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:45:01.437688 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431323 4931 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:45:01.437688 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431327 4931 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:45:01.437688 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431332 4931 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:45:01.437688 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431338 4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:45:01.437688 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431342 4931 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:45:01.437688 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431346 4931 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:45:01.437688 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431361 4931 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:45:01.437688 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431367 4931 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:45:01.437688 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431374 4931 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:45:01.437688 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431380 4931 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:45:01.437688 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431385 4931 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:45:01.437688 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431392 4931 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:45:01.437688 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431396 4931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:45:01.438524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431401 4931 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:45:01.438524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431405 4931 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:45:01.438524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431409 4931 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:45:01.438524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431414 4931 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:45:01.438524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431418 4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:45:01.438524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431422 4931 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:45:01.438524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431426 4931 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:45:01.438524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431433 4931 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:45:01.438524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431437 4931 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:45:01.438524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431442 4931 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:45:01.438524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431447 4931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:45:01.438524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431451 4931 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:45:01.438524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431455 4931 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:45:01.438524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431460 4931 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:45:01.438524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431464 4931 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:45:01.438524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431468 4931 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:45:01.438524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431472 4931 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:45:01.438524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431477 4931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:45:01.438524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431481 4931 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:45:01.438524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431485 4931 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:45:01.439451 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431489 4931 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:45:01.439451 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431494 4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:45:01.439451 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431498 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:45:01.439451 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431502 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:45:01.439451 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431507 4931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:45:01.439451 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431511 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:45:01.439451 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431523 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:45:01.439451 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431527 4931 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:45:01.439451 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431531 4931 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:45:01.439451 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431535 4931 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:45:01.439451 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431540 4931 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:45:01.439451 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431544 4931 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:45:01.439451 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431549 4931 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:45:01.439451 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431553 4931 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:45:01.439451 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431559 4931 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:45:01.439451 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431565 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:45:01.439451 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431570 4931 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:45:01.439451 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431575 4931 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:45:01.439451 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431579 4931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:45:01.440313 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431585 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:45:01.440313 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431592 4931 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:45:01.440313 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431596 4931 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:45:01.440313 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431601 4931 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:45:01.440313 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.431605 4931 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:45:01.440313 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431743 4931 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 12:45:01.440313 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431759 4931 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 12:45:01.440313 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431772 4931 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 12:45:01.440313 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431779 4931 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 12:45:01.440313 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431785 4931 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 12:45:01.440313 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431790 4931 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 12:45:01.440313 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431797 4931 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 12:45:01.440313 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431803 4931 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 12:45:01.440313 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431809 4931 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 12:45:01.440313 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431814 4931 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 12:45:01.440313 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431819 4931 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 12:45:01.440313 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431824 4931 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 12:45:01.440313 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431829 4931 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 12:45:01.440313 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431834 4931 flags.go:64] FLAG: --cgroup-root="" Oct 30 12:45:01.440313 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431839 4931 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 12:45:01.440313 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431845 4931 flags.go:64] FLAG: --client-ca-file="" Oct 30 12:45:01.440313 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431850 4931 flags.go:64] FLAG: --cloud-provider="external" Oct 30 12:45:01.440313 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431855 4931 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 12:45:01.441390 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431862 4931 flags.go:64] FLAG: --cluster-domain="" Oct 30 12:45:01.441390 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431867 4931 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 12:45:01.441390 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431872 4931 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 12:45:01.441390 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431877 4931 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 12:45:01.441390 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431882 4931 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 12:45:01.441390 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431889 4931 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 12:45:01.441390 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431894 4931 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 12:45:01.441390 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431899 4931 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 12:45:01.441390 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431905 4931 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 12:45:01.441390 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431912 4931 flags.go:64] FLAG: --contention-profiling="false" Oct 30 12:45:01.441390 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431916 4931 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 12:45:01.441390 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431921 4931 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 12:45:01.441390 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431926 4931 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 12:45:01.441390 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431931 4931 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 12:45:01.441390 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431938 4931 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 12:45:01.441390 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431944 4931 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 12:45:01.441390 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431948 4931 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 12:45:01.441390 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431953 4931 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 12:45:01.441390 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431958 4931 flags.go:64] FLAG: --enable-server="true" Oct 30 12:45:01.441390 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431963 4931 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 12:45:01.441390 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431969 4931 flags.go:64] FLAG: --event-burst="100" Oct 30 12:45:01.441390 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431974 4931 flags.go:64] FLAG: --event-qps="50" Oct 30 12:45:01.441390 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431979 4931 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 12:45:01.441390 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431984 4931 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 12:45:01.442525 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431989 4931 flags.go:64] FLAG: --eviction-hard="" Oct 30 12:45:01.442525 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.431995 4931 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 12:45:01.442525 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432000 4931 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 12:45:01.442525 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432005 4931 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 12:45:01.442525 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432010 4931 flags.go:64] FLAG: --eviction-soft="" Oct 30 12:45:01.442525 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432014 4931 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 12:45:01.442525 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432020 4931 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 12:45:01.442525 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432025 4931 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 12:45:01.442525 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432030 4931 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 12:45:01.442525 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432035 4931 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 12:45:01.442525 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432040 4931 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 12:45:01.442525 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432044 4931 flags.go:64] FLAG: --feature-gates="" Oct 30 12:45:01.442525 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432051 4931 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 12:45:01.442525 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432055 4931 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 12:45:01.442525 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432060 4931 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 12:45:01.442525 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432065 4931 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 12:45:01.442525 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432070 4931 flags.go:64] FLAG: --healthz-port="10248" Oct 30 12:45:01.442525 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432080 4931 flags.go:64] FLAG: --help="false" Oct 30 12:45:01.442525 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432086 4931 flags.go:64] FLAG: --hostname-override="ip-10-0-142-194.ec2.internal" Oct 30 12:45:01.442525 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432091 4931 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 12:45:01.442525 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432095 4931 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 12:45:01.442525 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432100 4931 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 12:45:01.442525 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432106 4931 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 12:45:01.442525 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432112 4931 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 12:45:01.443558 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432117 4931 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 12:45:01.443558 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432121 4931 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 12:45:01.443558 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432126 4931 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 12:45:01.443558 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432131 4931 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 12:45:01.443558 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432135 4931 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 12:45:01.443558 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432140 4931 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 12:45:01.443558 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432145 4931 flags.go:64] FLAG: --kube-reserved="" Oct 30 12:45:01.443558 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432150 4931 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 12:45:01.443558 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432155 4931 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 12:45:01.443558 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432160 4931 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 12:45:01.443558 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432165 4931 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 12:45:01.443558 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432170 4931 flags.go:64] FLAG: --lock-file="" Oct 30 12:45:01.443558 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432174 4931 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 12:45:01.443558 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432179 4931 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 12:45:01.443558 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432184 4931 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 12:45:01.443558 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432195 4931 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 12:45:01.443558 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432200 4931 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 12:45:01.443558 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432205 4931 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 12:45:01.443558 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432210 4931 flags.go:64] FLAG: --logging-format="text" Oct 30 12:45:01.443558 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432215 4931 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 12:45:01.443558 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432220 4931 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 12:45:01.443558 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432225 4931 flags.go:64] FLAG: --manifest-url="" Oct 30 12:45:01.443558 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432229 4931 flags.go:64] FLAG: --manifest-url-header="" Oct 30 12:45:01.443558 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432236 4931 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 12:45:01.443558 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432246 4931 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 12:45:01.444688 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432254 4931 flags.go:64] FLAG: --max-pods="110" Oct 30 12:45:01.444688 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432259 4931 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 12:45:01.444688 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432280 4931 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 12:45:01.444688 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432285 4931 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 12:45:01.444688 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432289 4931 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 12:45:01.444688 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432295 4931 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 12:45:01.444688 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432300 4931 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 12:45:01.444688 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432304 4931 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 12:45:01.444688 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432320 4931 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 12:45:01.444688 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432325 4931 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 12:45:01.444688 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432330 4931 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 12:45:01.444688 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432335 4931 flags.go:64] FLAG: --pod-cidr="" Oct 30 12:45:01.444688 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432339 4931 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 12:45:01.444688 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432344 4931 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 12:45:01.444688 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432356 4931 flags.go:64] FLAG: --pods-per-core="0" Oct 30 12:45:01.444688 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432360 4931 flags.go:64] FLAG: --port="10250" Oct 30 12:45:01.444688 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432365 4931 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 12:45:01.444688 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432370 4931 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-035fb9a9971d8c7ef" Oct 30 12:45:01.444688 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432375 4931 flags.go:64] FLAG: --qos-reserved="" Oct 30 12:45:01.444688 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432380 4931 flags.go:64] FLAG: --read-only-port="10255" Oct 30 12:45:01.444688 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432384 4931 flags.go:64] FLAG: --register-node="true" Oct 30 12:45:01.444688 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432389 4931 flags.go:64] FLAG: --register-with-taints="" Oct 30 12:45:01.444688 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432397 4931 flags.go:64] FLAG: --registry-burst="10" Oct 30 12:45:01.444688 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432403 4931 flags.go:64] FLAG: --registry-qps="5" Oct 30 12:45:01.444688 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432408 4931 flags.go:64] FLAG: --reserved-cpus="" Oct 30 12:45:01.444688 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432413 4931 flags.go:64] FLAG: --reserved-memory="" Oct 30 12:45:01.445805 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432419 4931 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 12:45:01.445805 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432424 4931 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 12:45:01.445805 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432429 4931 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 12:45:01.445805 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432433 4931 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 12:45:01.445805 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432438 4931 flags.go:64] FLAG: --runonce="false" Oct 30 12:45:01.445805 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432443 4931 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 12:45:01.445805 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432450 4931 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 12:45:01.445805 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432455 4931 flags.go:64] FLAG: --seccomp-default="false" Oct 30 12:45:01.445805 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432460 4931 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 12:45:01.445805 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432465 4931 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 12:45:01.445805 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432470 4931 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 12:45:01.445805 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432475 4931 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 12:45:01.445805 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432480 4931 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 12:45:01.445805 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432484 4931 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 12:45:01.445805 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432489 4931 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 12:45:01.445805 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432494 4931 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 12:45:01.445805 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432498 4931 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 12:45:01.445805 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432503 4931 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 12:45:01.445805 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432508 4931 flags.go:64] FLAG: --system-cgroups="" Oct 30 12:45:01.445805 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432513 4931 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 12:45:01.445805 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432522 4931 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 12:45:01.445805 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432526 4931 flags.go:64] FLAG: --tls-cert-file="" Oct 30 12:45:01.445805 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432531 4931 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 12:45:01.445805 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432539 4931 flags.go:64] FLAG: --tls-min-version="" Oct 30 12:45:01.445805 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432543 4931 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 12:45:01.446561 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432549 4931 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 12:45:01.446561 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432554 4931 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 12:45:01.446561 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432559 4931 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 12:45:01.446561 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432565 4931 flags.go:64] FLAG: --v="2" Oct 30 12:45:01.446561 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432577 4931 flags.go:64] FLAG: --version="false" Oct 30 12:45:01.446561 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432584 4931 flags.go:64] FLAG: --vmodule="" Oct 30 12:45:01.446561 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432591 4931 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 12:45:01.446561 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.432596 4931 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 12:45:01.446561 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432817 4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:45:01.446561 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432824 4931 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:45:01.446561 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432829 4931 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:45:01.446561 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432834 4931 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:45:01.446561 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432839 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:45:01.446561 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432846 4931 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:45:01.446561 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432851 4931 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:45:01.446561 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432856 4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:45:01.446561 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432860 4931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:45:01.446561 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432865 4931 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:45:01.446561 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432869 4931 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:45:01.446561 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432873 4931 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:45:01.446561 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432878 4931 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:45:01.446561 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432882 4931 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:45:01.447232 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432887 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:45:01.447232 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432891 4931 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:45:01.447232 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432896 4931 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:45:01.447232 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432900 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:45:01.447232 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432904 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:45:01.447232 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432909 4931 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:45:01.447232 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432913 4931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:45:01.447232 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432917 4931 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:45:01.447232 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432922 4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:45:01.447232 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432927 4931 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:45:01.447232 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432931 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:45:01.447232 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432936 4931 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:45:01.447232 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432940 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:45:01.447232 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432946 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:45:01.447232 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432950 4931 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:45:01.447232 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432954 4931 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:45:01.447232 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432960 4931 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:45:01.447232 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432964 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:45:01.447232 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432969 4931 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:45:01.448064 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432973 4931 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:45:01.448064 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432978 4931 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:45:01.448064 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432982 4931 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:45:01.448064 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432986 4931 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:45:01.448064 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432992 4931 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:45:01.448064 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.432996 4931 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:45:01.448064 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433001 4931 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:45:01.448064 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433006 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:45:01.448064 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433010 4931 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:45:01.448064 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433015 4931 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:45:01.448064 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433019 4931 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:45:01.448064 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433023 4931 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:45:01.448064 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433027 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:45:01.448064 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433032 4931 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:45:01.448064 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433036 4931 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:45:01.448064 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433040 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:45:01.448064 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433044 4931 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:45:01.448064 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433049 4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:45:01.448064 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433053 4931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:45:01.448064 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433058 4931 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:45:01.448875 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433062 4931 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:45:01.448875 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433066 4931 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:45:01.448875 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433071 4931 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:45:01.448875 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433077 4931 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:45:01.448875 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433083 4931 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:45:01.448875 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433088 4931 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:45:01.448875 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433095 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:45:01.448875 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433100 4931 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:45:01.448875 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433104 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:45:01.448875 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433109 4931 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:45:01.448875 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433114 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:45:01.448875 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433118 4931 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:45:01.448875 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433123 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:45:01.448875 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433127 4931 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:45:01.448875 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433131 4931 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:45:01.448875 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433135 4931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:45:01.448875 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433142 4931 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:45:01.448875 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433147 4931 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:45:01.448875 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433151 4931 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:45:01.449521 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433155 4931 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:45:01.449521 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433160 4931 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:45:01.449521 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433164 4931 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:45:01.449521 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433168 4931 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:45:01.449521 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433173 4931 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:45:01.449521 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433177 4931 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:45:01.449521 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433181 4931 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:45:01.449521 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433186 4931 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:45:01.449521 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433192 4931 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:45:01.449521 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433200 4931 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:45:01.449521 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433204 4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:45:01.449521 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433209 4931 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:45:01.449521 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433213 4931 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:45:01.449521 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433217 4931 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:45:01.449521 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433221 4931 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:45:01.449521 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433225 4931 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:45:01.449521 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433229 4931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:45:01.449521 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433234 4931 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:45:01.449521 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433238 4931 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:45:01.449521 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433244 4931 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:45:01.450038 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433248 4931 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:45:01.450038 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433252 4931 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:45:01.450038 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433257 4931 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:45:01.450038 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433276 4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:45:01.450038 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433282 4931 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:45:01.450038 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433287 4931 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:45:01.450038 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433291 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:45:01.450038 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433295 4931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:45:01.450038 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433300 4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:45:01.450038 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433306 4931 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:45:01.450038 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433310 4931 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:45:01.450038 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433314 4931 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:45:01.450038 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433318 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:45:01.450038 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433323 4931 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:45:01.450038 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.433365 4931 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:45:01.450038 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433418 4931 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:45:01.450452 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433427 4931 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:45:01.450452 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433433 4931 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:45:01.450452 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433438 4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:45:01.450452 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433443 4931 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:45:01.450452 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433447 4931 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:45:01.450452 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433451 4931 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:45:01.450452 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433456 4931 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:45:01.450452 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433461 4931 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:45:01.450452 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433465 4931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:45:01.450452 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433469 4931 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:45:01.450452 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433474 4931 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:45:01.450452 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433478 4931 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:45:01.450452 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433482 4931 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:45:01.450452 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433487 4931 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:45:01.450452 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433491 4931 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:45:01.450452 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433496 4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:45:01.450452 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433503 4931 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:45:01.450452 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433507 4931 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:45:01.450452 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433512 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:45:01.450920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433516 4931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:45:01.450920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433521 4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:45:01.450920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433526 4931 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:45:01.450920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433530 4931 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:45:01.450920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433534 4931 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:45:01.450920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433539 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:45:01.450920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433545 4931 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:45:01.450920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433549 4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:45:01.450920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433554 4931 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:45:01.450920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433558 4931 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:45:01.450920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433563 4931 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:45:01.450920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433567 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:45:01.450920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433571 4931 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:45:01.450920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433576 4931 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:45:01.450920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433580 4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:45:01.450920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433585 4931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:45:01.450920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433589 4931 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:45:01.450920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433593 4931 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:45:01.450920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433598 4931 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:45:01.450920 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433602 4931 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:45:01.451426 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433606 4931 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:45:01.451426 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433611 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:45:01.451426 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433615 4931 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:45:01.451426 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433620 4931 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:45:01.451426 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433624 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:45:01.451426 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433629 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:45:01.451426 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433633 4931 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:45:01.451426 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433637 4931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:45:01.451426 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433641 4931 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:45:01.451426 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433646 4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:45:01.451426 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433651 4931 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:45:01.451426 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433655 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:45:01.451426 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433660 4931 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:45:01.451426 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433665 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:45:01.451426 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433669 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:45:01.451426 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433675 4931 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:45:01.451426 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433680 4931 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:45:01.451426 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433685 4931 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:45:01.451426 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433691 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:45:01.451909 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433695 4931 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:45:01.451909 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433700 4931 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:45:01.451909 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433704 4931 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:45:01.451909 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433708 4931 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:45:01.451909 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433713 4931 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:45:01.451909 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433719 4931 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:45:01.451909 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433724 4931 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:45:01.451909 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433729 4931 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:45:01.451909 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433733 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:45:01.451909 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433738 4931 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:45:01.451909 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433742 4931 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:45:01.451909 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433746 4931 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:45:01.451909 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433750 4931 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:45:01.451909 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433755 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:45:01.451909 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433759 4931 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:45:01.451909 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433763 4931 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:45:01.451909 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433767 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:45:01.451909 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433772 4931 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:45:01.451909 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433777 4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:45:01.451909 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433781 4931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:45:01.452611 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433786 4931 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:45:01.452611 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433790 4931 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:45:01.452611 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433796 4931 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:45:01.452611 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433800 4931 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:45:01.452611 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433807 4931 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:45:01.452611 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433814 4931 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:45:01.452611 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433820 4931 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:45:01.452611 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433824 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:45:01.452611 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433829 4931 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:45:01.452611 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433834 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:45:01.452611 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433839 4931 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:45:01.452611 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433845 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:45:01.452611 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433860 4931 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:45:01.452611 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433865 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:45:01.452611 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433869 4931 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:45:01.452611 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433875 4931 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:45:01.452611 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433879 4931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:45:01.452611 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433883 4931 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:45:01.452611 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433888 4931 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:45:01.453066 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433892 4931 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:45:01.453066 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433896 4931 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:45:01.453066 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433901 4931 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:45:01.453066 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433905 4931 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:45:01.453066 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433909 4931 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:45:01.453066 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433914 4931 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:45:01.453066 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433918 4931 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:45:01.453066 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.433925 4931 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:45:01.453066 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.439192 4931 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 12:45:01.453066 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.439212 4931 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 12:45:01.453066 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.439236 4931 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 12:45:01.453066 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.439243 4931 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 12:45:01.453066 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439351 4931 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:45:01.453066 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439360 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:45:01.453066 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439366 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:45:01.453066 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439372 4931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:45:01.453066 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439377 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:45:01.453066 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439382 4931 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:45:01.453066 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439386 4931 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:45:01.453561 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439391 4931 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:45:01.453561 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439395 4931 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:45:01.453561 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439400 4931 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:45:01.453561 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439404 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:45:01.453561 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439408 4931 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:45:01.453561 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439413 4931 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:45:01.453561 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439418 4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:45:01.453561 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439422 4931 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:45:01.453561 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439427 4931 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:45:01.453561 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439431 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:45:01.453561 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439435 4931 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:45:01.453561 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439440 4931 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:45:01.453561 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439444 4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:45:01.453561 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439448 4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:45:01.453561 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439453 4931 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:45:01.453561 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439457 4931 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:45:01.453561 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439461 4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:45:01.453561 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439465 4931 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:45:01.453561 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439469 4931 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:45:01.453561 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439473 4931 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:45:01.454046 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439478 4931 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:45:01.454046 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439483 4931 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:45:01.454046 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439487 4931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:45:01.454046 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439492 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:45:01.454046 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439496 4931 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:45:01.454046 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439500 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:45:01.454046 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439504 4931 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:45:01.454046 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439509 4931 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:45:01.454046 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439513 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:45:01.454046 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439518 4931 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:45:01.454046 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439522 4931 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:45:01.454046 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439528 4931 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:45:01.454046 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439535 4931 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:45:01.454046 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439540 4931 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:45:01.454046 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439544 4931 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:45:01.454046 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439548 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:45:01.454046 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439553 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:45:01.454046 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439558 4931 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:45:01.454046 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439562 4931 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:45:01.454524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439568 4931 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:45:01.454524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439572 4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:45:01.454524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439577 4931 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:45:01.454524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439581 4931 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:45:01.454524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439585 4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:45:01.454524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439590 4931 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:45:01.454524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439594 4931 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:45:01.454524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439598 4931 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:45:01.454524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439603 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:45:01.454524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439607 4931 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:45:01.454524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439611 4931 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:45:01.454524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439616 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:45:01.454524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439620 4931 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:45:01.454524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439624 4931 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:45:01.454524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439629 4931 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:45:01.454524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439636 4931 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:45:01.454524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439642 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:45:01.454524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439647 4931 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:45:01.454524 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439652 4931 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:45:01.454977 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439657 4931 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:45:01.454977 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439662 4931 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:45:01.454977 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439666 4931 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:45:01.454977 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439671 4931 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:45:01.454977 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439676 4931 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:45:01.454977 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439680 4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:45:01.454977 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439684 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:45:01.454977 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439688 4931 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:45:01.454977 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439693 4931 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:45:01.454977 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439697 4931 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:45:01.454977 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439701 4931 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:45:01.454977 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439705 4931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:45:01.454977 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439710 4931 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:45:01.454977 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439714 4931 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:45:01.454977 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439719 4931 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:45:01.454977 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439723 4931 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:45:01.454977 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439728 4931 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:45:01.454977 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439733 4931 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:45:01.454977 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439737 4931 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:45:01.455449 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439741 4931 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:45:01.455449 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439745 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:45:01.455449 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439749 4931 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:45:01.455449 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439754 4931 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:45:01.455449 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439758 4931 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:45:01.455449 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439763 4931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:45:01.455449 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439767 4931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:45:01.455449 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439771 4931 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:45:01.455449 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439775 4931 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:45:01.455449 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439779 4931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:45:01.455449 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439784 4931 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:45:01.455449 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439788 4931 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:45:01.455449 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439792 4931 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:45:01.455449 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439797 4931 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:45:01.455449 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439801 4931 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:45:01.455449 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439806 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:45:01.455449 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439810 4931 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:45:01.455449 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439814 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:45:01.455449 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439818 4931 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:45:01.455449 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439823 4931 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:45:01.455949 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439827 4931 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:45:01.455949 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.439831 4931 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:45:01.455949 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.439873 4931 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:45:01.455949 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440099 4931 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:45:01.455949 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440107 4931 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:45:01.455949 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440111 4931 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:45:01.455949 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440116 4931 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:45:01.455949 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440120 4931 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:45:01.455949 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440129 4931 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:45:01.455949 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440134 4931 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:45:01.455949 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440139 4931 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:45:01.455949 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440143 4931 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:45:01.455949 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440148 4931 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:45:01.455949 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440152 4931 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:45:01.455949 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440157 4931 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:45:01.455949 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440161 4931 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:45:01.456595 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440165 4931 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:45:01.456595 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440169 4931 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:45:01.456595 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440174 4931 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:45:01.456595 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440178 4931 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:45:01.456595 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440183 4931 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:45:01.456595 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440187 4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:45:01.456595 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440191 4931 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:45:01.456595 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440196 4931 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:45:01.456595 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440201 4931 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:45:01.456595 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440208 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:45:01.456595 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440212 4931 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:45:01.456595 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440216 4931 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:45:01.456595 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440221 4931 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:45:01.456595 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440225 4931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:45:01.456595 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440230 4931 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:45:01.456595 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440235 4931 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:45:01.456595 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440239 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:45:01.456595 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440244 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:45:01.456595 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440248 4931 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:45:01.456595 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440253 4931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:45:01.457408 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440257 4931 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:45:01.457408 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440278 4931 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:45:01.457408 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440284 4931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:45:01.457408 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440289 4931 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:45:01.457408 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440293 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:45:01.457408 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440298 4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:45:01.457408 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440303 4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:45:01.457408 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440307 4931 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:45:01.457408 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440311 4931 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:45:01.457408 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440316 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:45:01.457408 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440320 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:45:01.457408 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440324 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:45:01.457408 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440328 4931 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:45:01.457408 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440332 4931 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:45:01.457408 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440337 4931 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:45:01.457408 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440341 4931 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:45:01.457408 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440345 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:45:01.457408 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440349 4931 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:45:01.457408 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440353 4931 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:45:01.457408 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440358 4931 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:45:01.457962 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440363 4931 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:45:01.457962 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440367 4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:45:01.457962 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440371 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:45:01.457962 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440375 4931 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:45:01.457962 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440380 4931 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:45:01.457962 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440384 4931 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:45:01.457962 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440390 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:45:01.457962 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440395 4931 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:45:01.457962 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440400 4931 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:45:01.457962 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440404 4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:45:01.457962 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440409 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:45:01.457962 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440413 4931 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:45:01.457962 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440419 4931 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:45:01.457962 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440426 4931 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:45:01.457962 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440432 4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:45:01.457962 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440437 4931 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:45:01.457962 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440441 4931 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:45:01.457962 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440446 4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:45:01.457962 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440452 4931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:45:01.458446 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440456 4931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:45:01.458446 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440461 4931 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:45:01.458446 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440465 4931 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:45:01.458446 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440470 4931 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:45:01.458446 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440474 4931 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:45:01.458446 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440479 4931 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:45:01.458446 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440483 4931 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:45:01.458446 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440488 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:45:01.458446 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440492 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:45:01.458446 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440496 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:45:01.458446 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440501 4931 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:45:01.458446 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440505 4931 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:45:01.458446 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440509 4931 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:45:01.458446 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440514 4931 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:45:01.458446 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440518 4931 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:45:01.458446 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440522 4931 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:45:01.458446 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440527 4931 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:45:01.458446 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440531 4931 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:45:01.458446 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440535 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:45:01.458446 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440542 4931 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:45:01.458981 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440551 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:45:01.458981 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440556 4931 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:45:01.458981 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440562 4931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:45:01.458981 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440566 4931 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:45:01.458981 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440571 4931 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:45:01.458981 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440575 4931 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:45:01.458981 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440579 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:45:01.458981 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440584 4931 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:45:01.458981 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440588 4931 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:45:01.458981 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440593 4931 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:45:01.458981 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440597 4931 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:45:01.458981 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440601 4931 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:45:01.458981 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440606 4931 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:45:01.458981 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440611 4931 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:45:01.458981 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.440652 4931 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:45:01.458981 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440707 4931 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:45:01.459390 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440715 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:45:01.459390 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440721 4931 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:45:01.459390 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440726 4931 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:45:01.459390 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440731 4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:45:01.459390 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440735 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:45:01.459390 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440740 4931 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:45:01.459390 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440746 4931 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:45:01.459390 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440751 4931 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:45:01.459390 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440756 4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:45:01.459390 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440760 4931 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:45:01.459390 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440764 4931 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:45:01.459390 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440769 4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:45:01.459390 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440773 4931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:45:01.459390 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440777 4931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:45:01.459390 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440782 4931 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:45:01.459390 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440787 4931 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:45:01.459390 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440793 4931 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:45:01.459390 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440798 4931 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:45:01.459390 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440802 4931 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:45:01.459865 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440806 4931 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:45:01.459865 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440811 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:45:01.459865 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440815 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:45:01.459865 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440820 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:45:01.459865 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440824 4931 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:45:01.459865 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440828 4931 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:45:01.459865 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440833 4931 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:45:01.459865 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440837 4931 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:45:01.459865 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440841 4931 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:45:01.459865 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440846 4931 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:45:01.459865 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440851 4931 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:45:01.459865 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440855 4931 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:45:01.459865 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440860 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:45:01.459865 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440868 4931 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:45:01.459865 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440874 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:45:01.459865 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440879 4931 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:45:01.459865 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440883 4931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:45:01.459865 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440888 4931 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:45:01.459865 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440892 4931 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:45:01.460334 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440897 4931 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:45:01.460334 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440902 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:45:01.460334 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440907 4931 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:45:01.460334 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440911 4931 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:45:01.460334 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440915 4931 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:45:01.460334 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440920 4931 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:45:01.460334 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440924 4931 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:45:01.460334 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440928 4931 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:45:01.460334 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440933 4931 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:45:01.460334 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440937 4931 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:45:01.460334 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440944 4931 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:45:01.460334 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440948 4931 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:45:01.460334 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440952 4931 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:45:01.460334 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440956 4931 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:45:01.460334 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440961 4931 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:45:01.460334 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440966 4931 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:45:01.460334 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440970 4931 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:45:01.460334 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440975 4931 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:45:01.460334 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440979 4931 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:45:01.460334 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440984 4931 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:45:01.460855 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440988 4931 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:45:01.460855 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440993 4931 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:45:01.460855 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.440997 4931 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:45:01.460855 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441001 4931 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:45:01.460855 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441007 4931 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:45:01.460855 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441011 4931 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:45:01.460855 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441016 4931 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:45:01.460855 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441020 4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:45:01.460855 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441025 4931 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:45:01.460855 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441029 4931 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:45:01.460855 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441033 4931 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:45:01.460855 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441038 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:45:01.460855 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441042 4931 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:45:01.460855 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441050 4931 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:45:01.460855 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441055 4931 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:45:01.460855 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441060 4931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:45:01.460855 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441064 4931 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:45:01.460855 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441068 4931 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:45:01.460855 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441073 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:45:01.460855 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441077 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:45:01.461681 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441081 4931 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:45:01.461681 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441086 4931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:45:01.461681 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441092 4931 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:45:01.461681 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441096 4931 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:45:01.461681 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441101 4931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:45:01.461681 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441105 4931 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:45:01.461681 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441110 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:45:01.461681 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441114 4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:45:01.461681 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441119 4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:45:01.461681 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441123 4931 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:45:01.461681 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441127 4931 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:45:01.461681 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441132 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:45:01.461681 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441137 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:45:01.461681 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441141 4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:45:01.461681 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441145 4931 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:45:01.461681 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441150 4931 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:45:01.461681 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441154 4931 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:45:01.461681 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441159 4931 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:45:01.461681 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441164 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:45:01.462593 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441168 4931 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:45:01.462593 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441172 4931 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:45:01.462593 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441179 4931 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:45:01.462593 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441183 4931 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:45:01.462593 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441188 4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:45:01.462593 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441192 4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:45:01.462593 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441198 4931 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:45:01.462593 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:01.441203 4931 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:45:01.462593 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.441298 4931 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 12:45:01.462593 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.442779 4931 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 12:45:01.462593 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.442852 4931 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 12:45:01.462593 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.443115 4931 server.go:1014] "Starting client certificate rotation" Oct 30 12:45:01.462593 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.443248 4931 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 12:45:01.462593 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.443306 4931 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 12:39:20 +0000 UTC" deadline="2028-06-02 22:21:37.330416383 +0000 UTC" Oct 30 12:45:01.462593 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.443333 4931 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13953h36m35.887084906s" Oct 30 12:45:01.462593 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.443838 4931 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 12:45:01.463297 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.445476 4931 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 12:45:01.463297 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.446733 4931 log.go:25] "Validated CRI v1 runtime API" Oct 30 12:45:01.463297 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.448822 4931 log.go:25] "Validated CRI v1 image API" Oct 30 12:45:01.463297 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.449452 4931 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 12:45:01.463297 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.451197 4931 fs.go:135] Filesystem UUIDs: map[1de3ac00-83a9-42cb-83d5-8961964fc912:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b2c69896-e7a2-41a5-abde-6b98e042ad92:/dev/nvme0n1p4] Oct 30 12:45:01.463473 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.451211 4931 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0653b6460c0da538c8b59897aba69293f877ed4a6073a6228e50cc5d07cb7fd0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0653b6460c0da538c8b59897aba69293f877ed4a6073a6228e50cc5d07cb7fd0/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0ab47e3edb68621f3c234d249753548ecf5793b8b8f7e6979f58a69f451b244a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0ab47e3edb68621f3c234d249753548ecf5793b8b8f7e6979f58a69f451b244a/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/115ad062fe20e4014413c9535f123fd48eb68e3ede1918a04b12d1dde106cc8f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/115ad062fe20e4014413c9535f123fd48eb68e3ede1918a04b12d1dde106cc8f/userdata/shm major:0 minor:321 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1b6dccf7539753c2a0a42bbe3427f9581938d66159b1837a9871a2072ba8ad44/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1b6dccf7539753c2a0a42bbe3427f9581938d66159b1837a9871a2072ba8ad44/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/44d0e397fc4ff49927726cc881f7e74c7f036d0707f71c992e117b6f7a99ed79/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/44d0e397fc4ff49927726cc881f7e74c7f036d0707f71c992e117b6f7a99ed79/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b184290d081eb9feb51fc83eaab0b1960f37c98985e35ce272e3603ef62b343/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b184290d081eb9feb51fc83eaab0b1960f37c98985e35ce272e3603ef62b343/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/508dfb09a0efe309cd0e4d6bad5cb15be4499a215e5928ba834d4e93372773d9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/508dfb09a0efe309cd0e4d6bad5cb15be4499a215e5928ba834d4e93372773d9/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/532538a4de233945df82f5a57d2553398c6ca40852be60528ea13b458fb46270/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/532538a4de233945df82f5a57d2553398c6ca40852be60528ea13b458fb46270/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/55b9ad4e2e989f2262058882b22a435d407b6cbde7e0eecc3efbb3662b5c036f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/55b9ad4e2e989f2262058882b22a435d407b6cbde7e0eecc3efbb3662b5c036f/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b1db096190d30041623d13c301ebc223aa7dd6b6f4a2e6c5d8c83d8ddc2b25e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b1db096190d30041623d13c301ebc223aa7dd6b6f4a2e6c5d8c83d8ddc2b25e/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e12ab82540a054fe54f452c4a8daa219be67c5c24fe14221a6d11d74b209d0f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e12ab82540a054fe54f452c4a8daa219be67c5c24fe14221a6d11d74b209d0f/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7525ad76683e03ad57ea0c28ae15ecaf1c8b91efedf5c3c5076bb832aab1e008/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7525ad76683e03ad57ea0c28ae15ecaf1c8b91efedf5c3c5076bb832aab1e008/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7a483e1e8ed42ec26ba81aa23d28315642ff2d3290e48d62cadb2abcf16dfbf6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7a483e1e8ed42ec26ba81aa23d28315642ff2d3290e48d62cadb2abcf16dfbf6/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/83241f9dbf1834814e3c065fb1d7d6086c72b4794461d53f9db893b020aa409e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/83241f9dbf1834814e3c065fb1d7d6086c72b4794461d53f9db893b020aa409e/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a7780b600c563f8797c365efb4bce1595f38972c8931e6286bbe49c4596ba076/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a7780b600c563f8797c365efb4bce1595f38972c8931e6286bbe49c4596ba076/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b216ecfdfe602b17806dca9305b70e41094ae20770a00460efa7dc75c45854bc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b216ecfdfe602b17806dca9305b70e41094ae20770a00460efa7dc75c45854bc/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b6d636bf80c4c19da1db3aa6ee2fe68380d6a29b597a7941fcee2644cef8b287/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b6d636bf80c4c19da1db3aa6ee2fe68380d6a29b597a7941fcee2644cef8b287/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c782a8370078036e2519d5d3f1e041769ae869568ed6a1a8f23b119f9a26daf3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c782a8370078036e2519d5d3f1e041769ae869568ed6a1a8f23b119f9a26daf3/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d450f953eb8dafb942c3939eddfdcca7a77b9b9d54e838fd3c3c588b8fc84f5d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d450f953eb8dafb942c3939eddfdcca7a77b9b9d54e838fd3c3c588b8fc84f5d/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d5af4ca60da37a49e6d2fb7625c05a2773f1bbecd91127113ce4562dd50da318/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d5af4ca60da37a49e6d2fb7625c05a2773f1bbecd91127113ce4562dd50da318/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dad2b65cdfd7add6edfe6d49ac05b02f31d74f0d50ae71f853de7dc5771538e1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dad2b65cdfd7add6edfe6d49ac05b02f31d74f0d50ae71f853de7dc5771538e1/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e45d0bc7ec0cb1a81c9a35075a9a3ad86e1483cf5806ee6d6e1f6a57709967a7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e45d0bc7ec0cb1a81c9a35075a9a3ad86e1483cf5806ee6d6e1f6a57709967a7/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e4d8508e11cecca7ed2dbb5583af705e3979519f1ca6335b0bf42e686eddf99c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e4d8508e11cecca7ed2dbb5583af705e3979519f1ca6335b0bf42e686eddf99c/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00ee6aae-7264-4c8d-a337-c7c9f1132e9c/volumes/kubernetes.io~projected/kube-api-access-wtcpz:{mountpoint:/var/lib/kubelet/pods/00ee6aae-7264-4c8d-a337-c7c9f1132e9c/volumes/kubernetes.io~projected/kube-api-access-wtcpz major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00ee6aae-7264-4c8d-a337-c7c9f1132e9c/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/00ee6aae-7264-4c8d-a337-c7c9f1132e9c/volumes/kubernetes.io~secret/serving-cert major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0/volumes/kubernetes.io~projected/kube-api-access-2szrp:{mountpoint:/var/lib/kubelet/pods/1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0/volumes/kubernetes.io~projected/kube-api-access-2szrp major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a6ee59d-3b62-4be1-87e8-79bb3bece069/volumes/kubernetes.io~projected/kube-api-access-gs4g4:{mountpoint:/var/lib/kubelet/pods/1a6ee59d-3b62-4be1-87e8-79bb3bece069/volumes/kubernetes.io~projected/kube-api-access-gs4g4 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d06e291-b9d3-48f7-8fc2-f615440631d9/volumes/kubernetes.io~projected/kube-api-access-jpzms:{mountpoint:/var/lib/kubelet/pods/1d06e291-b9d3-48f7-8fc2-f615440631d9/volumes/kubernetes.io~projected/kube-api-access-jpzms major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d06e291-b9d3-48f7-8fc2-f615440631d9/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/1d06e291-b9d3-48f7-8fc2-f615440631d9/volumes/kubernetes.io~secret/default-certificate major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d06e291-b9d3-48f7-8fc2-f615440631d9/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/1d06e291-b9d3-48f7-8fc2-f615440631d9/volumes/kubernetes.io~secret/stats-auth major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21ea018f-70e6-4ee3-952d-704089532abb/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/21ea018f-70e6-4ee3-952d-704089532abb/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24a00317-8fb9-41f3-8e1a-d1d985887725/volumes/kubernetes.io~projected/kube-api-access-lpmfr:{mountpoint:/var/lib/kubelet/pods/24a00317-8fb9-41f3-8e1a-d1d985887725/volumes/kubernetes.io~projected/kube-api-access-lpmfr major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37620bdd-11a4-4c12-8be1-9813a85d575b/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/37620bdd-11a4-4c12-8be1-9813a85d575b/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40168acf-dcfa-4323-91eb-e710a2b82caa/volumes/kubernetes.io~projected/kube-api-access-j94kk:{mountpoint:/var/lib/kubelet/pods/40168acf-dcfa-4323-91eb-e710a2b82caa/volumes/kubernetes.io~projected/kube-api-access-j94kk major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49c39196-4e77-4bcf-86b1-705a263e394b/volumes/kubernetes.io~projected/kube-api-access-sn2wd:{mountpoint:/var/lib/kubelet/pods/49c39196-4e77-4bcf-86b1-705a263e394b/volumes/kubernetes.io~projected/kube-api-access-sn2wd major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c/volumes/kubernetes.io~projected/kube-api-access-4g248:{mountpoint:/var/lib/kubelet/pods/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c/volumes/kubernetes.io~projected/kube-api-access-4g248 major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4df70e16-5acb-494e-ab67-855c809e3411/volumes/kubernetes.io~projected/kube-api-access-hkdqk:{mountpoint:/var/lib/kubelet/pods/4df70e16-5acb-494e-ab67-855c809e3411/volumes/kubernetes.io~projected/kube-api-access-hkdqk major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4df70e16-5acb-494e-ab67-855c809e3411/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/4df70e16-5acb-494e-ab67-855c809e3411/volumes/kubernetes.io~secret/cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3/volumes/kubernetes.io~projected/kube-api-access-h92pg:{mountpoint:/var/lib/kubelet/pods/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3/volumes/kubernetes.io~projected/kube-api-access-h92pg major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6000e43a-e175-46fe-b5f7-a417fc1ba5bb/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/6000e43a-e175-46fe-b5f7-a417fc1ba5bb/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6000e43a-e175-46fe-b5f7-a417fc1ba5bb/volumes/kubernetes.io~projected/kube-api-access-zks6k:{mountpoint:/var/lib/kubelet/pods/6000e43a-e175-46fe-b5f7-a417fc1ba5bb/volumes/kubernetes.io~projected/kube-api-access-zks6k major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6000e43a-e175-46fe-b5f7-a417fc1ba5bb/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/6000e43a-e175-46fe-b5f7-a417fc1ba5bb/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6000e43a-e175-46fe-b5f7-a417fc1ba5bb/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/6000e43a-e175-46fe-b5f7-a417fc1ba5bb/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64bdbc16-e845-45da-8f34-f0d788a75eac/volumes/kubernetes.io~projected/kube-api-access-wppl9:{mountpoint:/var/lib/kubelet/pods/64bdbc16-e845-45da-8f34-f0d788a75eac/volumes/kubernetes.io~projected/kube-api-access-wppl9 major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71e82393-029b-4388-86df-8bd068720867/volumes/kubernetes.io~projected/kube-api-access-kbnzh:{mountpoint:/var/lib/kubelet/pods/71e82393-029b-4388-86df-8bd068720867/volumes/kubernetes.io~projected/kube-api-access-kbnzh major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71e82393-029b-4388-86df-8bd068720867/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/71e82393-029b-4388-86df-8bd068720867/volumes/kubernetes.io~secret/serving-cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5/volumes/kubernetes.io~projected/kube-api-access-wfff9:{mountpoint:/var/lib/kubelet/pods/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5/volumes/kubernetes.io~projected/kube-api-access-wfff9 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7664727a-bcd2-4169-b70b-a0ecba045321/volumes/kubernetes.io~projected/kube-api-access-vzptg:{mountpoint:/var/lib/kubelet/pods/7664727a-bcd2-4169-b70b-a0ecba045321/volumes/kubernetes.io~projected/kube-api-access-vzptg major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7cb39685-b929-48bd-afd2-228862e62668/volumes/kubernetes.io~projected/kube-api-access-p6xkw:{mountpoint:/var/lib/kubelet/pods/7cb39685-b929-48bd-afd2-228862e62668/volumes/kubernetes.io~projected/kube-api-access-p6xkw major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8107811a-ec7c-497a-83fc-372249074496/volumes/kubernetes.io~projected/kube-api-access-jpcwv:{mountpoint:/var/lib/kubelet/pods/8107811a-ec7c-497a-83fc-372249074496/volumes/kubernetes.io~projected/kube-api-access-jpcwv major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a050566-8e52-4ea5-a4b6-575b1b4c9548/volumes/kubernetes.io~projected/kube-api-access-v6rnd:{mountpoint:/var/lib/kubelet/pods/8a050566-8e52-4ea5-a4b6-575b1b4c9548/volumes/kubernetes.io~projected/kube-api-access-v6rnd major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93e1032c-06a8-4418-99b7-540d1a4f1232/volumes/kubernetes.io~projected/kube-api-access-p8wc5:{mountpoint:/var/lib/kubelet/pods/93e1032c-06a8-4418-99b7-540d1a4f1232/volumes/kubernetes.io~projected/kube-api-access-p8wc5 major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c377915-c344-4791-baf6-4b286b9c019b/volumes/kubernetes.io~projected/kube-api-access-mbwtv:{mountpoint:/var/lib/kubelet/pods/9c377915-c344-4791-baf6-4b286b9c019b/volumes/kubernetes.io~projected/kube-api-access-mbwtv major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3928331-ce67-4c37-9c5d-63b0da642a0f/volumes/kubernetes.io~projected/kube-api-access-vf8zf:{mountpoint:/var/lib/kubelet/pods/a3928331-ce67-4c37-9c5d-63b0da642a0f/volumes/kubernetes.io~projected/kube-api-access-vf8zf major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae9fe0c2-0d00-4891-8549-bc99f5373da9/volumes/kubernetes.io~projected/kube-api-access-pnz9b:{mountpoint:/var/lib/kubelet/pods/ae9fe0c2-0d00-4891-8549-bc99f5373da9/volumes/kubernetes.io~projected/kube-api-access-pnz9b major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2bdf343-b9cd-484a-a8c1-1d0713d74093/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/b2bdf343-b9cd-484a-a8c1-1d0713d74093/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2bdf343-b9cd-484a-a8c1-1d0713d74093/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/b2bdf343-b9cd-484a-a8c1-1d0713d74093/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2bdf343-b9cd-484a-a8c1-1d0713d74093/volumes/kubernetes.io~projected/kube-api-access-cbk56:{mountpoint:/var/lib/kubelet/pods/b2bdf343-b9cd-484a-a8c1-1d0713d74093/volumes/kubernetes.io~projected/kube-api-access-cbk56 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b32002db-049b-4456-91d7-66c33a85c040/volumes/kubernetes.io~projected/kube-api-access-pdkm8:{mountpoint:/var/lib/kubelet/pods/b32002db-049b-4456-91d7-66c33a85c040/volumes/kubernetes.io~projected/kube-api-access-pdkm8 major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc5eaee0-b01e-432b-86ae-0d97a04da650/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/bc5eaee0-b01e-432b-86ae-0d97a04da650/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc5eaee0-b01e-432b-86ae-0d97a04da650/volumes/kubernetes.io~projected/kube-api-access-5dkqx:{mountpoint:/var/lib/kubelet/pods/bc5eaee0-b01e-432b-86ae-0d97a04da650/volumes/kubernetes.io~projected/kube-api-access-5dkqx major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc5eaee0-b01e-432b-86ae-0d97a04da650/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/bc5eaee0-b01e-432b-86ae-0d97a04da650/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5/volumes/kubernetes.io~projected/kube-api-access-jwpj2:{mountpoint:/var/lib/kubelet/pods/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5/volumes/kubernetes.io~projected/kube-api-access-jwpj2 major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e4bf0a41-a4bd-47ad-93b1-088500499acb/volumes/kubernetes.io~projected/kube-api-access-lcpmg:{mountpoint:/var/lib/kubelet/pods/e4bf0a41-a4bd-47ad-93b1-088500499acb/volumes/kubernetes.io~projected/kube-api-access-lcpmg major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e4bf0a41-a4bd-47ad-93b1-088500499acb/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/e4bf0a41-a4bd-47ad-93b1-088500499acb/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb4ec7a3-2fd7-44dd-98ca-72710972a050/volumes/kubernetes.io~projected/kube-api-access-j4rqz:{mountpoint:/var/lib/kubelet/pods/eb4ec7a3-2fd7-44dd-98ca-72710972a050/volumes/kubernetes.io~projected/kube-api-access-j4rqz major:0 minor:293 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/bf1c08076fc51c432282e79be743bd81a533d3c10f8cd7ae524ed43cc925f136/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/d7b862519ded6a1e92ba7182da0e25af0eb9a2da65e01b2abe65e0dbb914ecd9/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/0a881bcc4b9bc086e653648cec87fbca3fd8f8f33a4f90782c8fae3f39f439f4/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/86ce8145f620f332c6b0e6b45298ca4970244c4f29e3c4c7b53d0a3ac132bedf/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/a49259a591c428c9ad1b05426676bc54c40248d9c88d8e83f3a06acc8914418e/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/af1c98c5c4382c2803c6901aafc21021a528b2b0b9c223739daf54dd98f38a60/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/11232ae43b2d80d3c372bb90b76f0ce7055460cf3e2657abfdb5c448215a5120/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/b6a2e20a6600f05c4676541a1b86f0689d746afd447449e557d26d2ff3061fcf/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/cd3f926390bc11fa989f8573b8f51301b06798a6bc1b7edff28708da99b12728/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/25b382af4ae666ca2a3c7eb7044f571468d91a4e34a3d00ebedb5c6b1548773f/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/9b91db866b3ec066234cf770cecdccca7850b9361742cfcf11f472641644e26c/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/03ef194428220135787b64c211f7cb1f32f8f351430a0751b760978d53ce888c/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/b778786b3983fa9d30f47191ea2db1b9a3de1f19b53a0f9b64316e5d88fb05fb/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/023ffb5bf54716690a7b11bb0efc60ab22c7f4105ca895dd5c2f0e844099ba08/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/7f7e944f4d48ebd341aeed027b4234abeca27e05138146c201d05f67368ff2e8/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/01a92c49f08af3605cb4416263825a7bcecbba13bc381bb120906107e18c9e4e/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/a511b5d8204f63a0cc5410fb78e330527b8a2620d88beff8076de36e9c823279/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/176fb9f8d829aeba7c5c9e63c0dbea78f689615ec6472063b8dbf12076696275/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/4c248d95dddee832faec8c694ebd7be9a447287343dd5ddbb97a47dd5550f18c/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-184:{mountpoint:/var/lib/containers/storage/overlay/71f786a587032d788dca28ea00606fb88df7120b64b50d746fcb3130d335cab0/merged major:0 minor:184 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/c03a322a39608c5be53af5cad7d3bc975f69529ba6f24a224d3fcbc2f5d7ee99/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/c197f02332ae222197248c5395283c28ba126417a86fc1ab19b350f51a89fba7/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/8b57bf6bbcf2fd3626a14efa31412f1320b421d4e0dee01ba1fe03dc99088b17/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/49b2751ad31da3708eeff59e2d6764a9413e48975667ba5ea793a8b227a755a9/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/f244379b2abafc9e8b34d01dd3644246c6b5ad7493d3e6136870243bf4d02986/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/b26a0ca82f7a4246e497889736ce8d1ef2e62bcae5aa6f870878553a9457fe36/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/0fdcc8787d6c193dab7bf1461d3f64b01d0db0bc70d88ec06539fc158169f559/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/9c12bc08abcbbac9c36cbf303c58cf4341d0564aa50fae36335e5bb38fbdd010/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/cec63dfd8ab6f17e335a2d428fd36fadd55bbdf99e4ac1413a5cdefc05690354/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/bd5e9a453275b99d4db3f77137b3f4780c52f15d5e8c94185868734d27cf2636/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/88f5d5170e2c603b0ba62f3ba0e41fca3f7d2188c5100204fb7f538b345eb898/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/4601b68aadbcbf532c79ad3669a803661100364bd9dba639b655b5dfdda3a89d/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/a6ec296f332d21c5524eb7e7d8fa1f98965ad9457ff082c3c5a7f7d012f12e4a/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/7a5abe9861c21e49552a10853818a976768e2c997e02659243a5c02f738d7493/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/97b79dd53891f2f94adf356f01e04d9d2ecbd13ae2298c2922b8f4401edb9f96/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/5ed7259133f3ad06ac7c44ca8feaef1a0d157d88ad243b6342dc2d1466989b8b/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/1e81d8b6a1455f7ed060bbb14b6a9e56112fb81d43b793b1930d24afd57d4b21/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/c49bd4da96c4ef24971fd262d0fd05180ac7c0265eac91d7269a9a7715069664/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/e06a91d4d186f8d8e0d4978889d970764b404905f6a4b0f6d1a068f0d813d24a/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/fba2841b685fc95ed47e6ba5bb236c3f3ce2d2e0c72469c7c4f754d4bb9be027/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/ddff962fcc83af6a6e9191e1b20735e1c1d3d2ece41b81c8de03cb47e88786e7/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/fcb2485460d838ada1ccdfdeaaedbfedd2644cb72b1d107fbb40ecd4dc16e5a1/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/85f04767225df6600f0e43f41a9c5b46e825cbce8d62daa48f06e72a845c5198/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/24eab20497d153e9352d16335f9ec43e0b3bbf891557837fa17e1473947650ff/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/b5e07b0d0188e9b49d50dac219021677ca27e28b37d0133dd4e6f21369e1e02d/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/d74069abf8975ed7dee7a5e1a0b2a5f5f64e0d317cdd29e8445e24586f819711/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/71595cb9aa693a63dc97a066049872653943a843064aa044168e631a7af5c5d0/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/07370d8b6f7cf7e919440214b23bb48181471a344194314b10addedc10dc8742/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/b92dfdd32fc757a1dcf68c31e75141b48f1bdde3a6bfe59d2e3693aaac4e4e22/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/ab2f5a531130381af1d45133c3ab533d95606674da76be7fac72d59b5cdd5dae/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/c9994d122e49f2e87370f795202f6e7b980332c6c0461f4cb4d26d3cce24bd19/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/07a5b48266698464cfe6825aa820b9a60ecae22563d556221e866381fd7c9c4a/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/9354e8bf0be57faa4b9d260b4f74033d7c2b9002dcf507f0612e689400dc0ca9/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-91:{mountpoint:/var/lib/containers/storage/overlay/0582eef86684cf437120024ccd74c8dd32c3b90c66d2b3f815ccc052b799edf0/merged major:0 minor:91 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/186bd916af6785191967b69edecbb2276c8944e67d16ba1b7ae4e5389ea1ae6a/merged major:0 minor:96 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/e08627c0ca17f6341ce0af7e394223f791a0a75087919c6255383d0c95a8c051/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 30 12:45:01.463687 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.460527 4931 manager.go:217] Machine: {Timestamp:2026-10-30 12:45:01.460056983 +0000 UTC m=+0.063555421 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3131040 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec228b8fd08c0c68e4e5dde1421ab5ea SystemUUID:ec228b8f-d08c-0c68-e4e5-dde1421ab5ea BootID:d46a3f6c-4b13-4bbb-a557-471599694e2e Filesystems:[{Device:overlay_0-91 DeviceMajor:0 DeviceMinor:91 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0ab47e3edb68621f3c234d249753548ecf5793b8b8f7e6979f58a69f451b244a/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/71e82393-029b-4388-86df-8bd068720867/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e4d8508e11cecca7ed2dbb5583af705e3979519f1ca6335b0bf42e686eddf99c/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e12ab82540a054fe54f452c4a8daa219be67c5c24fe14221a6d11d74b209d0f/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7525ad76683e03ad57ea0c28ae15ecaf1c8b91efedf5c3c5076bb832aab1e008/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6000e43a-e175-46fe-b5f7-a417fc1ba5bb/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40168acf-dcfa-4323-91eb-e710a2b82caa/volumes/kubernetes.io~projected/kube-api-access-j94kk DeviceMajor:0 DeviceMinor:312 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a3928331-ce67-4c37-9c5d-63b0da642a0f/volumes/kubernetes.io~projected/kube-api-access-vf8zf DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/71e82393-029b-4388-86df-8bd068720867/volumes/kubernetes.io~projected/kube-api-access-kbnzh DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/b2bdf343-b9cd-484a-a8c1-1d0713d74093/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ae9fe0c2-0d00-4891-8549-bc99f5373da9/volumes/kubernetes.io~projected/kube-api-access-pnz9b DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/00ee6aae-7264-4c8d-a337-c7c9f1132e9c/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6000e43a-e175-46fe-b5f7-a417fc1ba5bb/volumes/kubernetes.io~projected/kube-api-access-zks6k DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bc5eaee0-b01e-432b-86ae-0d97a04da650/volumes/kubernetes.io~projected/kube-api-access-5dkqx DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7664727a-bcd2-4169-b70b-a0ecba045321/volumes/kubernetes.io~projected/kube-api-access-vzptg DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9c377915-c344-4791-baf6-4b286b9c019b/volumes/kubernetes.io~projected/kube-api-access-mbwtv DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/64bdbc16-e845-45da-8f34-f0d788a75eac/volumes/kubernetes.io~projected/kube-api-access-wppl9 DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a7780b600c563f8797c365efb4bce1595f38972c8931e6286bbe49c4596ba076/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bc5eaee0-b01e-432b-86ae-0d97a04da650/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4df70e16-5acb-494e-ab67-855c809e3411/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c/volumes/kubernetes.io~projected/kube-api-access-4g248 DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b1db096190d30041623d13c301ebc223aa7dd6b6f4a2e6c5d8c83d8ddc2b25e/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/49c39196-4e77-4bcf-86b1-705a263e394b/volumes/kubernetes.io~projected/kube-api-access-sn2wd DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/44d0e397fc4ff49927726cc881f7e74c7f036d0707f71c992e117b6f7a99ed79/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b184290d081eb9feb51fc83eaab0b1960f37c98985e35ce272e3603ef62b343/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3/volumes/kubernetes.io~projected/kube-api-access-h92pg DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d5af4ca60da37a49e6d2fb7625c05a2773f1bbecd91127113ce4562dd50da318/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1d06e291-b9d3-48f7-8fc2-f615440631d9/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1b6dccf7539753c2a0a42bbe3427f9581938d66159b1837a9871a2072ba8ad44/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4df70e16-5acb-494e-ab67-855c809e3411/volumes/kubernetes.io~projected/kube-api-access-hkdqk DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dad2b65cdfd7add6edfe6d49ac05b02f31d74f0d50ae71f853de7dc5771538e1/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e4bf0a41-a4bd-47ad-93b1-088500499acb/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1d06e291-b9d3-48f7-8fc2-f615440631d9/volumes/kubernetes.io~projected/kube-api-access-jpzms DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5/volumes/kubernetes.io~projected/kube-api-access-wfff9 DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/532538a4de233945df82f5a57d2553398c6ca40852be60528ea13b458fb46270/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/37620bdd-11a4-4c12-8be1-9813a85d575b/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8107811a-ec7c-497a-83fc-372249074496/volumes/kubernetes.io~projected/kube-api-access-jpcwv DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/00ee6aae-7264-4c8d-a337-c7c9f1132e9c/volumes/kubernetes.io~projected/kube-api-access-wtcpz DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0/volumes/kubernetes.io~projected/kube-api-access-2szrp DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7cb39685-b929-48bd-afd2-228862e62668/volumes/kubernetes.io~projected/kube-api-access-p6xkw DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/21ea018f-70e6-4ee3-952d-704089532abb/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b216ecfdfe602b17806dca9305b70e41094ae20770a00460efa7dc75c45854bc/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/83241f9dbf1834814e3c065fb1d7d6086c72b4794461d53f9db893b020aa409e/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-184 DeviceMajor:0 DeviceMinor:184 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e4bf0a41-a4bd-47ad-93b1-088500499acb/volumes/kubernetes.io~projected/kube-api-access-lcpmg DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c782a8370078036e2519d5d3f1e041769ae869568ed6a1a8f23b119f9a26daf3/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b6d636bf80c4c19da1db3aa6ee2fe68380d6a29b597a7941fcee2644cef8b287/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1a6ee59d-3b62-4be1-87e8-79bb3bece069/volumes/kubernetes.io~projected/kube-api-access-gs4g4 DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb4ec7a3-2fd7-44dd-98ca-72710972a050/volumes/kubernetes.io~projected/kube-api-access-j4rqz DeviceMajor:0 DeviceMinor:293 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b32002db-049b-4456-91d7-66c33a85c040/volumes/kubernetes.io~projected/kube-api-access-pdkm8 DeviceMajor:0 DeviceMinor:319 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b2bdf343-b9cd-484a-a8c1-1d0713d74093/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b2bdf343-b9cd-484a-a8c1-1d0713d74093/volumes/kubernetes.io~projected/kube-api-access-cbk56 DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/115ad062fe20e4014413c9535f123fd48eb68e3ede1918a04b12d1dde106cc8f/userdata/shm DeviceMajor:0 DeviceMinor:321 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/24a00317-8fb9-41f3-8e1a-d1d985887725/volumes/kubernetes.io~projected/kube-api-access-lpmfr DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/508dfb09a0efe309cd0e4d6bad5cb15be4499a215e5928ba834d4e93372773d9/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/93e1032c-06a8-4418-99b7-540d1a4f1232/volumes/kubernetes.io~projected/kube-api-access-p8wc5 DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7a483e1e8ed42ec26ba81aa23d28315642ff2d3290e48d62cadb2abcf16dfbf6/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e45d0bc7ec0cb1a81c9a35075a9a3ad86e1483cf5806ee6d6e1f6a57709967a7/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6000e43a-e175-46fe-b5f7-a417fc1ba5bb/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5/volumes/kubernetes.io~projected/kube-api-access-jwpj2 DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a050566-8e52-4ea5-a4b6-575b1b4c9548/volumes/kubernetes.io~projected/kube-api-access-v6rnd DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bc5eaee0-b01e-432b-86ae-0d97a04da650/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0653b6460c0da538c8b59897aba69293f877ed4a6073a6228e50cc5d07cb7fd0/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6000e43a-e175-46fe-b5f7-a417fc1ba5bb/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/55b9ad4e2e989f2262058882b22a435d407b6cbde7e0eecc3efbb3662b5c036f/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1d06e291-b9d3-48f7-8fc2-f615440631d9/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d450f953eb8dafb942c3939eddfdcca7a77b9b9d54e838fd3c3c588b8fc84f5d/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0653b6460c0da53 MacAddress:da:b4:02:be:63:da Speed:10000 Mtu:8901} {Name:115ad062fe20e40 MacAddress:aa:d1:77:45:a6:8c Speed:10000 Mtu:8901} {Name:1b6dccf7539753c MacAddress:0e:5d:da:bd:af:8b Speed:10000 Mtu:8901} {Name:4b184290d081eb9 MacAddress:26:f7:01:82:b5:1a Speed:10000 Mtu:8901} {Name:532538a4de23394 MacAddress:1a:37:1f:dd:9e:b6 Speed:10000 Mtu:8901} {Name:5b1db096190d300 MacAddress:d6:d7:8a:ae:b3:c6 Speed:10000 Mtu:8901} {Name:7a483e1e8ed42ec MacAddress:8e:17:84:7e:1a:b5 Speed:10000 Mtu:8901} {Name:83241f9dbf18348 MacAddress:5e:c8:34:27:4b:36 Speed:10000 Mtu:8901} {Name:a7780b600c563f8 MacAddress:2e:0a:12:ea:b3:d8 Speed:10000 Mtu:8901} {Name:b216ecfdfe602b1 MacAddress:36:e1:cc:bd:7b:a2 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d2:13:67:4d Speed:0 Mtu:9001} {Name:br-int MacAddress:5e:ca:93:0b:53:75 Speed:0 Mtu:8901} {Name:c782a8370078036 MacAddress:3e:2b:f5:51:6d:dd Speed:10000 Mtu:8901} {Name:d450f953eb8dafb MacAddress:86:c0:23:ce:00:eb Speed:10000 Mtu:8901} {Name:e4d8508e11cecca MacAddress:c2:25:5c:dc:ec:b2 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d2:13:67:4d Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:2e:8c:d2:fb:dd:75 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:42:77:51:7b:0c:12 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 12:45:01.463687 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.460849 4931 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 12:45:01.463687 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.460889 4931 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 12:45:01.463687 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.461223 4931 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 12:45:01.463687 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.461260 4931 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-194.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 12:45:01.463687 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.461456 4931 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 12:45:01.463687 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.461469 4931 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 12:45:01.463687 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.461482 4931 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 12:45:01.463687 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.461509 4931 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 12:45:01.463687 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.461537 4931 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 12:45:01.463687 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.461745 4931 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 12:45:01.463687 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.461887 4931 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 12:45:01.463687 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.461932 4931 kubelet.go:493] "Attempting to sync node with API server" Oct 30 12:45:01.463687 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.461945 4931 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 12:45:01.463687 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.461958 4931 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 12:45:01.463687 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.461971 4931 kubelet.go:405] "Adding apiserver pod source" Oct 30 12:45:01.463687 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.461983 4931 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 12:45:01.463687 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.462887 4931 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 12:45:01.463687 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.462900 4931 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 12:45:01.463687 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.463351 4931 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 12:45:01.465886 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.465871 4931 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 12:45:01.465886 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.465885 4931 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 12:45:01.465986 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.465977 4931 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 12:45:01.466156 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.466146 4931 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 12:45:01.466186 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.466175 4931 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 12:45:01.466276 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.466257 4931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 12:45:01.466307 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.466287 4931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 12:45:01.466307 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.466295 4931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 12:45:01.466307 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.466301 4931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 12:45:01.466307 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.466306 4931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 12:45:01.466414 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.466313 4931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 12:45:01.466414 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.466319 4931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 12:45:01.466414 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.466325 4931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 12:45:01.466414 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.466331 4931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 12:45:01.466414 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.466337 4931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 12:45:01.466414 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.466345 4931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 12:45:01.466414 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.466354 4931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 12:45:01.466414 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.466377 4931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 12:45:01.466414 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.466383 4931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 12:45:01.469672 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.469660 4931 server.go:1263] "Started kubelet" Oct 30 12:45:01.469735 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.469711 4931 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 12:45:01.470126 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.469765 4931 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 12:45:01.470126 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.469870 4931 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 12:45:01.470261 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.470197 4931 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 12:45:01.470233 ip-10-0-142-194 systemd[1]: Started Kubernetes Kubelet. Oct 30 12:45:01.472663 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.472640 4931 server.go:317] "Adding debug handlers to kubelet server" Oct 30 12:45:01.476670 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.475858 4931 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 12:45:01.476670 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.475873 4931 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 12:45:01.476670 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.475916 4931 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:39:22 +0000 UTC" deadline="2028-04-05 07:29:52.529556369 +0000 UTC" Oct 30 12:45:01.476670 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.475934 4931 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12546h44m51.053626217s" Oct 30 12:45:01.476670 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.475967 4931 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 12:45:01.476670 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.475975 4931 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 12:45:01.476670 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.476073 4931 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 12:45:01.478207 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478176 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cb39685-b929-48bd-afd2-228862e62668" volumeName="kubernetes.io/configmap/7cb39685-b929-48bd-afd2-228862e62668-config-volume" seLinuxMountContext="" Oct 30 12:45:01.478207 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478205 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00ee6aae-7264-4c8d-a337-c7c9f1132e9c" volumeName="kubernetes.io/configmap/00ee6aae-7264-4c8d-a337-c7c9f1132e9c-config" seLinuxMountContext="" Oct 30 12:45:01.478369 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478218 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0" volumeName="kubernetes.io/projected/1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0-kube-api-access-2szrp" seLinuxMountContext="" Oct 30 12:45:01.478369 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478230 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a6ee59d-3b62-4be1-87e8-79bb3bece069" volumeName="kubernetes.io/configmap/1a6ee59d-3b62-4be1-87e8-79bb3bece069-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 12:45:01.478369 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478244 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6000e43a-e175-46fe-b5f7-a417fc1ba5bb" volumeName="kubernetes.io/empty-dir/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-ca-trust-extracted" seLinuxMountContext="" Oct 30 12:45:01.478369 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478256 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6000e43a-e175-46fe-b5f7-a417fc1ba5bb" volumeName="kubernetes.io/secret/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-image-registry-private-configuration" seLinuxMountContext="" Oct 30 12:45:01.478369 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478282 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73b3e3a7-93fa-49c2-a8f1-4188a22b99a5" volumeName="kubernetes.io/configmap/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-service-ca-bundle" seLinuxMountContext="" Oct 30 12:45:01.478369 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478306 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6000e43a-e175-46fe-b5f7-a417fc1ba5bb" volumeName="kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-bound-sa-token" seLinuxMountContext="" Oct 30 12:45:01.478369 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:01.478308 4931 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-194.ec2.internal\" not found" Oct 30 12:45:01.478369 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478319 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64bdbc16-e845-45da-8f34-f0d788a75eac" volumeName="kubernetes.io/configmap/64bdbc16-e845-45da-8f34-f0d788a75eac-telemetry-config" seLinuxMountContext="" Oct 30 12:45:01.478369 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478362 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc5eaee0-b01e-432b-86ae-0d97a04da650" volumeName="kubernetes.io/secret/bc5eaee0-b01e-432b-86ae-0d97a04da650-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 12:45:01.478740 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478386 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4bf0a41-a4bd-47ad-93b1-088500499acb" volumeName="kubernetes.io/secret/e4bf0a41-a4bd-47ad-93b1-088500499acb-serving-cert" seLinuxMountContext="" Oct 30 12:45:01.478740 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478400 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24a00317-8fb9-41f3-8e1a-d1d985887725" volumeName="kubernetes.io/projected/24a00317-8fb9-41f3-8e1a-d1d985887725-kube-api-access-lpmfr" seLinuxMountContext="" Oct 30 12:45:01.478740 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478413 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49c39196-4e77-4bcf-86b1-705a263e394b" volumeName="kubernetes.io/configmap/49c39196-4e77-4bcf-86b1-705a263e394b-serviceca" seLinuxMountContext="" Oct 30 12:45:01.478740 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478425 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64bdbc16-e845-45da-8f34-f0d788a75eac" volumeName="kubernetes.io/projected/64bdbc16-e845-45da-8f34-f0d788a75eac-kube-api-access-wppl9" seLinuxMountContext="" Oct 30 12:45:01.478740 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478440 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73b3e3a7-93fa-49c2-a8f1-4188a22b99a5" volumeName="kubernetes.io/projected/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-kube-api-access-wfff9" seLinuxMountContext="" Oct 30 12:45:01.478740 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478453 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37620bdd-11a4-4c12-8be1-9813a85d575b" volumeName="kubernetes.io/configmap/37620bdd-11a4-4c12-8be1-9813a85d575b-konnectivity-ca" seLinuxMountContext="" Oct 30 12:45:01.478740 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478464 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40168acf-dcfa-4323-91eb-e710a2b82caa" volumeName="kubernetes.io/projected/40168acf-dcfa-4323-91eb-e710a2b82caa-kube-api-access-j94kk" seLinuxMountContext="" Oct 30 12:45:01.478740 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478476 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e40c7ec-9b38-4f24-8d7b-5689f029c0a3" volumeName="kubernetes.io/projected/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-kube-api-access-h92pg" seLinuxMountContext="" Oct 30 12:45:01.478740 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478489 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8107811a-ec7c-497a-83fc-372249074496" volumeName="kubernetes.io/projected/8107811a-ec7c-497a-83fc-372249074496-kube-api-access-jpcwv" seLinuxMountContext="" Oct 30 12:45:01.478740 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478500 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a050566-8e52-4ea5-a4b6-575b1b4c9548" volumeName="kubernetes.io/configmap/8a050566-8e52-4ea5-a4b6-575b1b4c9548-multus-daemon-config" seLinuxMountContext="" Oct 30 12:45:01.478740 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478510 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b32002db-049b-4456-91d7-66c33a85c040" volumeName="kubernetes.io/projected/b32002db-049b-4456-91d7-66c33a85c040-kube-api-access-pdkm8" seLinuxMountContext="" Oct 30 12:45:01.478740 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478533 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc5eaee0-b01e-432b-86ae-0d97a04da650" volumeName="kubernetes.io/projected/bc5eaee0-b01e-432b-86ae-0d97a04da650-kube-api-access-5dkqx" seLinuxMountContext="" Oct 30 12:45:01.478740 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478544 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6000e43a-e175-46fe-b5f7-a417fc1ba5bb" volumeName="kubernetes.io/configmap/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-trusted-ca" seLinuxMountContext="" Oct 30 12:45:01.478740 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478555 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d06e291-b9d3-48f7-8fc2-f615440631d9" volumeName="kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-stats-auth" seLinuxMountContext="" Oct 30 12:45:01.478740 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478566 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71e82393-029b-4388-86df-8bd068720867" volumeName="kubernetes.io/secret/71e82393-029b-4388-86df-8bd068720867-serving-cert" seLinuxMountContext="" Oct 30 12:45:01.478740 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478577 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cb39685-b929-48bd-afd2-228862e62668" volumeName="kubernetes.io/empty-dir/7cb39685-b929-48bd-afd2-228862e62668-tmp-dir" seLinuxMountContext="" Oct 30 12:45:01.478740 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478588 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="877cfa0b-b1c6-4455-b232-b1ca558b5084" volumeName="kubernetes.io/configmap/877cfa0b-b1c6-4455-b232-b1ca558b5084-nginx-conf" seLinuxMountContext="" Oct 30 12:45:01.478740 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478599 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a050566-8e52-4ea5-a4b6-575b1b4c9548" volumeName="kubernetes.io/projected/8a050566-8e52-4ea5-a4b6-575b1b4c9548-kube-api-access-v6rnd" seLinuxMountContext="" Oct 30 12:45:01.478740 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478613 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5" volumeName="kubernetes.io/secret/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-installation-pull-secrets" seLinuxMountContext="" Oct 30 12:45:01.478740 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478624 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21ea018f-70e6-4ee3-952d-704089532abb" volumeName="kubernetes.io/secret/21ea018f-70e6-4ee3-952d-704089532abb-original-pull-secret" seLinuxMountContext="" Oct 30 12:45:01.478740 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478635 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6000e43a-e175-46fe-b5f7-a417fc1ba5bb" volumeName="kubernetes.io/secret/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-installation-pull-secrets" seLinuxMountContext="" Oct 30 12:45:01.478740 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478647 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73b3e3a7-93fa-49c2-a8f1-4188a22b99a5" volumeName="kubernetes.io/empty-dir/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-tmp" seLinuxMountContext="" Oct 30 12:45:01.478740 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478659 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc5eaee0-b01e-432b-86ae-0d97a04da650" volumeName="kubernetes.io/configmap/bc5eaee0-b01e-432b-86ae-0d97a04da650-env-overrides" seLinuxMountContext="" Oct 30 12:45:01.478740 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478670 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4bf0a41-a4bd-47ad-93b1-088500499acb" volumeName="kubernetes.io/configmap/e4bf0a41-a4bd-47ad-93b1-088500499acb-config" seLinuxMountContext="" Oct 30 12:45:01.478740 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478682 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a6ee59d-3b62-4be1-87e8-79bb3bece069" volumeName="kubernetes.io/configmap/1a6ee59d-3b62-4be1-87e8-79bb3bece069-cni-binary-copy" seLinuxMountContext="" Oct 30 12:45:01.478740 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478694 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40168acf-dcfa-4323-91eb-e710a2b82caa" volumeName="kubernetes.io/empty-dir/40168acf-dcfa-4323-91eb-e710a2b82caa-data-volume" seLinuxMountContext="" Oct 30 12:45:01.478740 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478705 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc5eaee0-b01e-432b-86ae-0d97a04da650" volumeName="kubernetes.io/configmap/bc5eaee0-b01e-432b-86ae-0d97a04da650-ovnkube-script-lib" seLinuxMountContext="" Oct 30 12:45:01.478740 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478727 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a6ee59d-3b62-4be1-87e8-79bb3bece069" volumeName="kubernetes.io/configmap/1a6ee59d-3b62-4be1-87e8-79bb3bece069-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 12:45:01.478740 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478740 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5" volumeName="kubernetes.io/empty-dir/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-ca-trust-extracted" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478761 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73b3e3a7-93fa-49c2-a8f1-4188a22b99a5" volumeName="kubernetes.io/secret/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-serving-cert" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478772 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00ee6aae-7264-4c8d-a337-c7c9f1132e9c" volumeName="kubernetes.io/secret/00ee6aae-7264-4c8d-a337-c7c9f1132e9c-serving-cert" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478784 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24a00317-8fb9-41f3-8e1a-d1d985887725" volumeName="kubernetes.io/empty-dir/24a00317-8fb9-41f3-8e1a-d1d985887725-tmp-dir" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478795 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3928331-ce67-4c37-9c5d-63b0da642a0f" volumeName="kubernetes.io/projected/a3928331-ce67-4c37-9c5d-63b0da642a0f-kube-api-access-vf8zf" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478809 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2bdf343-b9cd-484a-a8c1-1d0713d74093" volumeName="kubernetes.io/projected/b2bdf343-b9cd-484a-a8c1-1d0713d74093-kube-api-access-cbk56" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478820 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5" volumeName="kubernetes.io/configmap/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-trusted-ca" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478832 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d06e291-b9d3-48f7-8fc2-f615440631d9" volumeName="kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-default-certificate" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478843 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37620bdd-11a4-4c12-8be1-9813a85d575b" volumeName="kubernetes.io/secret/37620bdd-11a4-4c12-8be1-9813a85d575b-agent-certs" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478856 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c" volumeName="kubernetes.io/configmap/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-config" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478867 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73b3e3a7-93fa-49c2-a8f1-4188a22b99a5" volumeName="kubernetes.io/configmap/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-trusted-ca-bundle" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478878 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a050566-8e52-4ea5-a4b6-575b1b4c9548" volumeName="kubernetes.io/configmap/8a050566-8e52-4ea5-a4b6-575b1b4c9548-cni-binary-copy" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478889 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc5eaee0-b01e-432b-86ae-0d97a04da650" volumeName="kubernetes.io/configmap/bc5eaee0-b01e-432b-86ae-0d97a04da650-ovnkube-config" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478901 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4df70e16-5acb-494e-ab67-855c809e3411" volumeName="kubernetes.io/projected/4df70e16-5acb-494e-ab67-855c809e3411-kube-api-access-hkdqk" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478925 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cb39685-b929-48bd-afd2-228862e62668" volumeName="kubernetes.io/projected/7cb39685-b929-48bd-afd2-228862e62668-kube-api-access-p6xkw" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478937 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5" volumeName="kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-bound-sa-token" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478951 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb4ec7a3-2fd7-44dd-98ca-72710972a050" volumeName="kubernetes.io/projected/eb4ec7a3-2fd7-44dd-98ca-72710972a050-kube-api-access-j4rqz" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478964 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a6ee59d-3b62-4be1-87e8-79bb3bece069" volumeName="kubernetes.io/projected/1a6ee59d-3b62-4be1-87e8-79bb3bece069-kube-api-access-gs4g4" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478983 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c" volumeName="kubernetes.io/projected/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-kube-api-access-4g248" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.478994 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71e82393-029b-4388-86df-8bd068720867" volumeName="kubernetes.io/configmap/71e82393-029b-4388-86df-8bd068720867-config" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.479005 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c377915-c344-4791-baf6-4b286b9c019b" volumeName="kubernetes.io/projected/9c377915-c344-4791-baf6-4b286b9c019b-kube-api-access-mbwtv" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.479016 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2bdf343-b9cd-484a-a8c1-1d0713d74093" volumeName="kubernetes.io/empty-dir/b2bdf343-b9cd-484a-a8c1-1d0713d74093-tmp" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.479026 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4df70e16-5acb-494e-ab67-855c809e3411" volumeName="kubernetes.io/secret/4df70e16-5acb-494e-ab67-855c809e3411-cert" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.479037 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00ee6aae-7264-4c8d-a337-c7c9f1132e9c" volumeName="kubernetes.io/configmap/00ee6aae-7264-4c8d-a337-c7c9f1132e9c-trusted-ca" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.479048 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d06e291-b9d3-48f7-8fc2-f615440631d9" volumeName="kubernetes.io/projected/1d06e291-b9d3-48f7-8fc2-f615440631d9-kube-api-access-jpzms" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.479061 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71e82393-029b-4388-86df-8bd068720867" volumeName="kubernetes.io/projected/71e82393-029b-4388-86df-8bd068720867-kube-api-access-kbnzh" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.479071 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73b3e3a7-93fa-49c2-a8f1-4188a22b99a5" volumeName="kubernetes.io/empty-dir/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-snapshots" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.479083 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2bdf343-b9cd-484a-a8c1-1d0713d74093" volumeName="kubernetes.io/empty-dir/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-tuned" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.479094 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0" volumeName="kubernetes.io/configmap/1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0-iptables-alerter-script" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.479105 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5" volumeName="kubernetes.io/configmap/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-certificates" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.479117 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00ee6aae-7264-4c8d-a337-c7c9f1132e9c" volumeName="kubernetes.io/projected/00ee6aae-7264-4c8d-a337-c7c9f1132e9c-kube-api-access-wtcpz" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.479185 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49c39196-4e77-4bcf-86b1-705a263e394b" volumeName="kubernetes.io/projected/49c39196-4e77-4bcf-86b1-705a263e394b-kube-api-access-sn2wd" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.479198 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6000e43a-e175-46fe-b5f7-a417fc1ba5bb" volumeName="kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-kube-api-access-zks6k" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.479210 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5" volumeName="kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-kube-api-access-jwpj2" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.479221 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5" volumeName="kubernetes.io/secret/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-image-registry-private-configuration" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.479234 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6000e43a-e175-46fe-b5f7-a417fc1ba5bb" volumeName="kubernetes.io/configmap/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-certificates" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.479246 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93e1032c-06a8-4418-99b7-540d1a4f1232" volumeName="kubernetes.io/projected/93e1032c-06a8-4418-99b7-540d1a4f1232-kube-api-access-p8wc5" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.479258 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae9fe0c2-0d00-4891-8549-bc99f5373da9" volumeName="kubernetes.io/projected/ae9fe0c2-0d00-4891-8549-bc99f5373da9-kube-api-access-pnz9b" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.479310 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4bf0a41-a4bd-47ad-93b1-088500499acb" volumeName="kubernetes.io/projected/e4bf0a41-a4bd-47ad-93b1-088500499acb-kube-api-access-lcpmg" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.479322 4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7664727a-bcd2-4169-b70b-a0ecba045321" volumeName="kubernetes.io/projected/7664727a-bcd2-4169-b70b-a0ecba045321-kube-api-access-vzptg" seLinuxMountContext="" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.479334 4931 reconstruct.go:97] "Volume reconstruction finished" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.479342 4931 reconciler.go:29] "Reconciler: start to sync state" Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.479591 4931 factory.go:55] Registering systemd factory Oct 30 12:45:01.479834 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.479607 4931 factory.go:223] Registration of the systemd container factory successfully Oct 30 12:45:01.481502 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.481486 4931 factory.go:153] Registering CRI-O factory Oct 30 12:45:01.481502 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.481500 4931 factory.go:223] Registration of the crio container factory successfully Oct 30 12:45:01.481660 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.481539 4931 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 12:45:01.481660 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.481556 4931 factory.go:103] Registering Raw factory Oct 30 12:45:01.481660 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.481566 4931 manager.go:1199] Started watching for new ooms in manager Oct 30 12:45:01.481916 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.481903 4931 manager.go:319] Starting recovery of all containers Oct 30 12:45:01.494500 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.494476 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/1.log" Oct 30 12:45:01.494500 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.494486 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/1.log" Oct 30 12:45:01.503776 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.503717 4931 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 12:45:01.505040 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.505018 4931 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 12:45:01.505040 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.505039 4931 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 12:45:01.505307 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.505059 4931 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 12:45:01.505307 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:01.505104 4931 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 12:45:01.530443 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.530408 4931 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:45:01.532235 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.532211 4931 generic.go:356] "Generic (PLEG): container finished" podID="1a6ee59d-3b62-4be1-87e8-79bb3bece069" containerID="2ed953f87fcd3b80b810eb9dcde021018945996eeab2216505eac23135e243ed" exitCode=0 Oct 30 12:45:01.532235 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.532233 4931 generic.go:356] "Generic (PLEG): container finished" podID="1a6ee59d-3b62-4be1-87e8-79bb3bece069" containerID="accefaf68319d46a349c7010819cdfd9934694c0d0dc8b9e70e55eec0636e46b" exitCode=0 Oct 30 12:45:01.532396 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.532240 4931 generic.go:356] "Generic (PLEG): container finished" podID="1a6ee59d-3b62-4be1-87e8-79bb3bece069" containerID="7c50afd1feaa4bdcf85dba3a5c549594e249274f4cf1b91c105e3a92dc22b2e4" exitCode=0 Oct 30 12:45:01.532396 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.532248 4931 generic.go:356] "Generic (PLEG): container finished" podID="1a6ee59d-3b62-4be1-87e8-79bb3bece069" containerID="8c35adbba6ffbc98b99b7318e5767509a09bdabebfa1d1d40f5b9451a6cdc473" exitCode=0 Oct 30 12:45:01.532396 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.532253 4931 generic.go:356] "Generic (PLEG): container finished" podID="1a6ee59d-3b62-4be1-87e8-79bb3bece069" containerID="2ecd8df8a3c867cb553128cf9953f4255f2003deba3c9f7e6d086c2418f95ba5" exitCode=0 Oct 30 12:45:01.532396 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.532259 4931 generic.go:356] "Generic (PLEG): container finished" podID="1a6ee59d-3b62-4be1-87e8-79bb3bece069" containerID="4f8d43f2361effef849ec26f6315d3482590bede29bba8c92756fbee702e3ed4" exitCode=0 Oct 30 12:45:01.545081 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.545061 4931 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:45:01.545498 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.545471 4931 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:45:01.545585 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.545571 4931 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 12:45:01.545639 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.545618 4931 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:45:01.546006 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.545990 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/1.log" Oct 30 12:45:01.546080 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.546013 4931 generic.go:356] "Generic (PLEG): container finished" podID="00ee6aae-7264-4c8d-a337-c7c9f1132e9c" containerID="c6678b84146d9fb4ce33ee83e569d21905f75a2151e695f43279698e8e6c13a8" exitCode=255 Oct 30 12:45:01.552942 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.552916 4931 generic.go:356] "Generic (PLEG): container finished" podID="fba8e851cd321bfcfae1acf3abfda6d3" containerID="801ca3f13c3f6c860263fe7b1e89db1f2fda7feba3e5c33407112880fed75c8d" exitCode=0 Oct 30 12:45:01.573405 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.573360 4931 manager.go:324] Recovery completed Oct 30 12:45:01.594315 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.594297 4931 cpu_manager.go:226] "Starting" policy="none" Oct 30 12:45:01.594315 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.594311 4931 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 12:45:01.594447 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.594329 4931 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 12:45:01.594478 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.594444 4931 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 12:45:01.594478 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.594455 4931 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 12:45:01.594478 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.594470 4931 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 12:45:01.594478 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.594475 4931 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 12:45:01.594478 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.594480 4931 policy_none.go:50] "Start" Oct 30 12:45:01.594609 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.594489 4931 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 12:45:01.594609 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.594498 4931 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 12:45:01.596113 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.596095 4931 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 12:45:01.596113 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.596110 4931 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 12:45:01.596193 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.596116 4931 policy_none.go:44] "Start" Oct 30 12:45:01.600480 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.600453 4931 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 12:45:01.602990 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.602976 4931 manager.go:341] "Starting Device Plugin manager" Oct 30 12:45:01.603684 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:01.603002 4931 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 12:45:01.603728 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.603691 4931 server.go:87] "Starting device plugin registration server" Oct 30 12:45:01.604014 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.604002 4931 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 12:45:01.604044 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.604018 4931 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 12:45:01.604130 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.604109 4931 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 12:45:01.604281 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.604224 4931 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 12:45:01.604378 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.604294 4931 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 12:45:01.604378 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.604302 4931 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 12:45:01.604484 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.604350 4931 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T12:45:01.604258478Z","UUID":"1194a800-516a-4466-ab24-135d3513a57b","Handler":null,"Name":"","Endpoint":""} Oct 30 12:45:01.605589 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:01.605570 4931 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 12:45:01.605743 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.605715 4931 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-194.ec2.internal"] Oct 30 12:45:01.606099 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.606082 4931 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 12:45:01.606194 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.606117 4931 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 12:45:01.606393 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.606377 4931 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" Oct 30 12:45:01.606466 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.606419 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" event={"ID":"fba8e851cd321bfcfae1acf3abfda6d3","Type":"ContainerStarted","Data":"b9abd4dff098f60e20f324ee0fce69b015c3e10fb17c6dc0bcfe37658e02bb00"} Oct 30 12:45:01.606466 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.606461 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" event={"ID":"fba8e851cd321bfcfae1acf3abfda6d3","Type":"ContainerDied","Data":"801ca3f13c3f6c860263fe7b1e89db1f2fda7feba3e5c33407112880fed75c8d"} Oct 30 12:45:01.606584 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.606481 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" event={"ID":"fba8e851cd321bfcfae1acf3abfda6d3","Type":"ContainerStarted","Data":"5e12ab82540a054fe54f452c4a8daa219be67c5c24fe14221a6d11d74b209d0f"} Oct 30 12:45:01.606584 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.606498 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-194.ec2.internal" event={"ID":"7bc488b7d1d453086a471da9a46ef80c","Type":"ContainerStarted","Data":"fe9369a42fdc8549e6f5a0ea452c0730f33a4d478cb96fa84ed72cd3aec448b8"} Oct 30 12:45:01.606584 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.606516 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-194.ec2.internal" event={"ID":"7bc488b7d1d453086a471da9a46ef80c","Type":"ContainerStarted","Data":"b6d636bf80c4c19da1db3aa6ee2fe68380d6a29b597a7941fcee2644cef8b287"} Oct 30 12:45:01.607441 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.607327 4931 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-194.ec2.internal" Oct 30 12:45:01.635818 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.635799 4931 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 12:45:01.635896 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:01.635849 4931 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-194.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-194.ec2.internal" Oct 30 12:45:01.636184 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.636172 4931 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 12:45:01.636218 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:01.636204 4931 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" Oct 30 12:45:01.704851 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.704821 4931 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:45:01.706081 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.706060 4931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:45:01.706170 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.706092 4931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:45:01.706170 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.706103 4931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:45:01.706170 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.706141 4931 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-194.ec2.internal" Oct 30 12:45:01.744759 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.744738 4931 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-142-194.ec2.internal" Oct 30 12:45:01.744837 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.744790 4931 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-194.ec2.internal" Oct 30 12:45:01.747359 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.747336 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fba8e851cd321bfcfae1acf3abfda6d3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal\" (UID: \"fba8e851cd321bfcfae1acf3abfda6d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" Oct 30 12:45:01.747436 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.747367 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7bc488b7d1d453086a471da9a46ef80c-config\") pod \"kube-apiserver-proxy-ip-10-0-142-194.ec2.internal\" (UID: \"7bc488b7d1d453086a471da9a46ef80c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-194.ec2.internal" Oct 30 12:45:01.747436 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.747388 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fba8e851cd321bfcfae1acf3abfda6d3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal\" (UID: \"fba8e851cd321bfcfae1acf3abfda6d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" Oct 30 12:45:01.847555 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.847502 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7bc488b7d1d453086a471da9a46ef80c-config\") pod \"kube-apiserver-proxy-ip-10-0-142-194.ec2.internal\" (UID: \"7bc488b7d1d453086a471da9a46ef80c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-194.ec2.internal" Oct 30 12:45:01.847555 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.847527 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fba8e851cd321bfcfae1acf3abfda6d3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal\" (UID: \"fba8e851cd321bfcfae1acf3abfda6d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" Oct 30 12:45:01.847657 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.847566 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fba8e851cd321bfcfae1acf3abfda6d3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal\" (UID: \"fba8e851cd321bfcfae1acf3abfda6d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" Oct 30 12:45:01.847657 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.847602 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fba8e851cd321bfcfae1acf3abfda6d3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal\" (UID: \"fba8e851cd321bfcfae1acf3abfda6d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" Oct 30 12:45:01.847657 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.847603 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fba8e851cd321bfcfae1acf3abfda6d3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal\" (UID: \"fba8e851cd321bfcfae1acf3abfda6d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" Oct 30 12:45:01.847657 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:01.847603 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7bc488b7d1d453086a471da9a46ef80c-config\") pod \"kube-apiserver-proxy-ip-10-0-142-194.ec2.internal\" (UID: \"7bc488b7d1d453086a471da9a46ef80c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-194.ec2.internal" Oct 30 12:45:02.462303 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.462255 4931 apiserver.go:52] "Watching apiserver" Oct 30 12:45:02.521848 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.521820 4931 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 12:45:02.522687 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.522666 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-9tp7b","openshift-dns/node-resolver-gt8nx","openshift-monitoring/cluster-monitoring-operator-884576c57-vjdmx","openshift-image-registry/node-ca-rjhg6","openshift-image-registry/image-registry-7bff985b5d-qwb5l","kube-system/kas-connection-checker-685895c7c9-rnflh","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-s7c7s","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pptmg","openshift-multus/multus-r54rk","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-c8pgj","openshift-insights/insights-runtime-extractor-b94k6","openshift-kube-storage-version-migrator/migrator-66df5d496b-4h8hg","openshift-ovn-kubernetes/ovnkube-node-8rpk8","openshift-insights/insights-operator-7c4986d449-sztw6","openshift-console-operator/console-operator-7cd56b8dbc-v4mks","openshift-ingress-canary/ingress-canary-dnpgs","kube-system/kas-connection-checker-685895c7c9-6xnwh","kube-system/kas-connection-checker-685895c7c9-cv78r","openshift-multus/network-metrics-daemon-j6b2w","openshift-network-diagnostics/network-check-source-6f78d59f99-pkpds","kube-system/global-pull-secret-syncer-nk9fb","openshift-dns/dns-default-w52p8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal","openshift-image-registry/image-registry-596f5cc4c5-sgvjw","openshift-network-diagnostics/network-check-target-ljvqz","openshift-network-console/networking-console-plugin-86c478bb88-5bmsj","openshift-cluster-node-tuning-operator/tuned-js5rp","kube-system/konnectivity-agent-k2fjm","openshift-service-ca-operator/service-ca-operator-544d749c8c-kzhq7","openshift-multus/multus-additional-cni-plugins-l2fgb","kube-system/kube-apiserver-proxy-ip-10-0-142-194.ec2.internal","openshift-ingress/router-default-5db587c4bc-dttsw"] Oct 30 12:45:02.524150 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.524135 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjdmx" Oct 30 12:45:02.524688 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.524656 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:45:02.525061 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.525041 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-s7c7s" Oct 30 12:45:02.529164 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.529134 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:45:02.530706 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.530686 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-5bmsj" Oct 30 12:45:02.532285 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.532254 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:45:02.536194 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.536173 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:45:02.536787 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.536772 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w52p8" Oct 30 12:45:02.538644 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.538626 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:45:02.539319 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.539304 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 12:45:02.539319 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.539312 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 12:45:02.540173 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.540155 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 12:45:02.540240 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.540223 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 12:45:02.540557 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.540537 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 12:45:02.540610 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.540566 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 12:45:02.540610 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.540577 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 12:45:02.540684 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.540634 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 12:45:02.540684 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.540623 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 12:45:02.540907 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.540889 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 12:45:02.551782 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.551766 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 12:45:02.551855 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.551841 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 12:45:02.551994 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.551979 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 12:45:02.552447 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.552434 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 12:45:02.552639 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.552627 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 12:45:02.553123 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.553108 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 12:45:02.553231 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.553219 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 12:45:02.553262 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.553224 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 12:45:02.554273 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.554247 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:45:02.555357 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.555339 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:45:02.563679 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.563657 4931 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" Oct 30 12:45:02.568315 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.568228 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:45:02.568534 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.568368 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 12:45:02.568534 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.568450 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 12:45:02.568534 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.568491 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 12:45:02.568534 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.568499 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 12:45:02.568534 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.568494 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 12:45:02.568534 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.568521 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 12:45:02.568534 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.568494 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 12:45:02.568534 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.568522 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 12:45:02.568828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.568529 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:45:02.568828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.568561 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 12:45:02.568828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.568576 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 12:45:02.568828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.568570 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 12:45:02.568828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.568593 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 12:45:02.568828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.568600 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 12:45:02.568828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.568601 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 12:45:02.568828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.568609 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 12:45:02.568828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.568645 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 12:45:02.568828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.568660 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 12:45:02.568828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.568701 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 12:45:02.568828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.568743 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 12:45:02.568828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.568764 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 12:45:02.569607 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.569577 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-9zx7f\"" Oct 30 12:45:02.569688 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.569637 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-svz4t\"" Oct 30 12:45:02.569688 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.569644 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-trr9l\"" Oct 30 12:45:02.569688 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.569652 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xlfj2\"" Oct 30 12:45:02.569688 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.569656 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5fhhc\"" Oct 30 12:45:02.569688 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.569681 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6g9fh\"" Oct 30 12:45:02.569985 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.569691 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 12:45:02.569985 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.569700 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-l246t\"" Oct 30 12:45:02.569985 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.569702 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pglr2\"" Oct 30 12:45:02.569985 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.569718 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 12:45:02.569985 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.569740 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-b79n8\"" Oct 30 12:45:02.569985 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.569756 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 12:45:02.569985 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.569760 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mwttw\"" Oct 30 12:45:02.569985 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.569781 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6f4z2\"" Oct 30 12:45:02.569985 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.569791 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 12:45:02.569985 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.569807 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-w75sr\"" Oct 30 12:45:02.569985 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.569813 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 12:45:02.569985 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.569741 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8b8c8\"" Oct 30 12:45:02.569985 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.569814 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g92fm\"" Oct 30 12:45:02.569985 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.569843 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 12:45:02.569985 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.569886 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mxwzk\"" Oct 30 12:45:02.569985 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.569886 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-k9vxw\"" Oct 30 12:45:02.569985 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.569929 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zngfb\"" Oct 30 12:45:02.569985 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.569939 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 12:45:02.569985 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.569949 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:45:02.569985 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.569961 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:45:02.569985 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.569982 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 12:45:02.569985 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.569996 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 12:45:02.570867 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.570013 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 12:45:02.570867 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.570031 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 12:45:02.570867 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.570064 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 12:45:02.570867 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.570067 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:45:02.570867 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.570074 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 12:45:02.570867 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.570113 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-k9mqg\"" Oct 30 12:45:02.570867 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.570121 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 12:45:02.570867 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.570153 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 12:45:02.570867 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.570154 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 12:45:02.570867 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.570191 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 12:45:02.570867 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.570222 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 12:45:02.570867 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.570261 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 12:45:02.570867 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.570446 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 12:45:02.570867 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.570470 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 12:45:02.570867 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.570628 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 12:45:02.570867 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.570648 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 12:45:02.570867 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.570704 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 12:45:02.570867 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.570871 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 12:45:02.571417 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.570884 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 12:45:02.571417 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.570887 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 12:45:02.571417 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.570905 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 12:45:02.571417 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.570924 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 12:45:02.571417 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.570940 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 12:45:02.571417 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.570945 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 12:45:02.571417 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.570967 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 12:45:02.571417 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.571002 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 12:45:02.571417 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.571014 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 12:45:02.571417 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.571050 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 12:45:02.571417 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.571093 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 12:45:02.571417 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.571156 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 12:45:02.578683 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.578668 4931 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 12:45:02.589972 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.589956 4931 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 12:45:02.590047 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:02.589998 4931 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" Oct 30 12:45:02.651673 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.651641 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/00ee6aae-7264-4c8d-a337-c7c9f1132e9c-serving-cert\") pod \"console-operator-7cd56b8dbc-v4mks\" (UID: \"00ee6aae-7264-4c8d-a337-c7c9f1132e9c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" Oct 30 12:45:02.651673 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.651678 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1a6ee59d-3b62-4be1-87e8-79bb3bece069-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:45:02.651817 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.651699 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0-host-slash\") pod \"iptables-alerter-9tp7b\" (UID: \"1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0\") " pod="openshift-network-operator/iptables-alerter-9tp7b" Oct 30 12:45:02.651817 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.651715 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-etc-openvswitch\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.651817 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.651756 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-system-cni-dir\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.651817 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.651784 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-var-lib-cni-multus\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.651817 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.651809 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbnzh\" (UniqueName: \"kubernetes.io/projected/71e82393-029b-4388-86df-8bd068720867-kube-api-access-kbnzh\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pptmg\" (UID: \"71e82393-029b-4388-86df-8bd068720867\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pptmg" Oct 30 12:45:02.652052 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.651881 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/00ee6aae-7264-4c8d-a337-c7c9f1132e9c-config\") pod \"console-operator-7cd56b8dbc-v4mks\" (UID: \"00ee6aae-7264-4c8d-a337-c7c9f1132e9c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" Oct 30 12:45:02.652052 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.651919 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-serving-cert\") pod \"insights-operator-7c4986d449-sztw6\" (UID: \"73b3e3a7-93fa-49c2-a8f1-4188a22b99a5\") " pod="openshift-insights/insights-operator-7c4986d449-sztw6" Oct 30 12:45:02.652052 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.651949 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e4bf0a41-a4bd-47ad-93b1-088500499acb-config\") pod \"service-ca-operator-544d749c8c-kzhq7\" (UID: \"e4bf0a41-a4bd-47ad-93b1-088500499acb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzhq7" Oct 30 12:45:02.652052 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.651975 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-systemd-units\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.652052 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652001 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4rqz\" (UniqueName: \"kubernetes.io/projected/eb4ec7a3-2fd7-44dd-98ca-72710972a050-kube-api-access-j4rqz\") pod \"network-check-target-ljvqz\" (UID: \"eb4ec7a3-2fd7-44dd-98ca-72710972a050\") " pod="openshift-network-diagnostics/network-check-target-ljvqz" Oct 30 12:45:02.652052 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652022 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/00ee6aae-7264-4c8d-a337-c7c9f1132e9c-serving-cert\") pod \"console-operator-7cd56b8dbc-v4mks\" (UID: \"00ee6aae-7264-4c8d-a337-c7c9f1132e9c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" Oct 30 12:45:02.652052 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652049 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfff9\" (UniqueName: \"kubernetes.io/projected/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-kube-api-access-wfff9\") pod \"insights-operator-7c4986d449-sztw6\" (UID: \"73b3e3a7-93fa-49c2-a8f1-4188a22b99a5\") " pod="openshift-insights/insights-operator-7c4986d449-sztw6" Oct 30 12:45:02.652399 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652060 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1a6ee59d-3b62-4be1-87e8-79bb3bece069-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:45:02.652399 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652076 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a6ee59d-3b62-4be1-87e8-79bb3bece069-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:45:02.652399 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652126 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-run-multus-certs\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.652399 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652159 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-serving-cert\") pod \"insights-operator-7c4986d449-sztw6\" (UID: \"73b3e3a7-93fa-49c2-a8f1-4188a22b99a5\") " pod="openshift-insights/insights-operator-7c4986d449-sztw6" Oct 30 12:45:02.652399 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652159 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/00ee6aae-7264-4c8d-a337-c7c9f1132e9c-config\") pod \"console-operator-7cd56b8dbc-v4mks\" (UID: \"00ee6aae-7264-4c8d-a337-c7c9f1132e9c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" Oct 30 12:45:02.652399 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652160 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/37620bdd-11a4-4c12-8be1-9813a85d575b-agent-certs\") pod \"konnectivity-agent-k2fjm\" (UID: \"37620bdd-11a4-4c12-8be1-9813a85d575b\") " pod="kube-system/konnectivity-agent-k2fjm" Oct 30 12:45:02.652399 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652185 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e4bf0a41-a4bd-47ad-93b1-088500499acb-config\") pod \"service-ca-operator-544d749c8c-kzhq7\" (UID: \"e4bf0a41-a4bd-47ad-93b1-088500499acb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzhq7" Oct 30 12:45:02.652399 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652224 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtcpz\" (UniqueName: \"kubernetes.io/projected/00ee6aae-7264-4c8d-a337-c7c9f1132e9c-kube-api-access-wtcpz\") pod \"console-operator-7cd56b8dbc-v4mks\" (UID: \"00ee6aae-7264-4c8d-a337-c7c9f1132e9c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" Oct 30 12:45:02.652399 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652253 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-lib-modules\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.652763 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652397 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf8zf\" (UniqueName: \"kubernetes.io/projected/a3928331-ce67-4c37-9c5d-63b0da642a0f-kube-api-access-vf8zf\") pod \"kas-connection-checker-685895c7c9-rnflh\" (UID: \"a3928331-ce67-4c37-9c5d-63b0da642a0f\") " pod="kube-system/kas-connection-checker-685895c7c9-rnflh" Oct 30 12:45:02.652763 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652432 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-snapshots\") pod \"insights-operator-7c4986d449-sztw6\" (UID: \"73b3e3a7-93fa-49c2-a8f1-4188a22b99a5\") " pod="openshift-insights/insights-operator-7c4986d449-sztw6" Oct 30 12:45:02.652763 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652468 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-sztw6\" (UID: \"73b3e3a7-93fa-49c2-a8f1-4188a22b99a5\") " pod="openshift-insights/insights-operator-7c4986d449-sztw6" Oct 30 12:45:02.652763 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652476 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/37620bdd-11a4-4c12-8be1-9813a85d575b-agent-certs\") pod \"konnectivity-agent-k2fjm\" (UID: \"37620bdd-11a4-4c12-8be1-9813a85d575b\") " pod="kube-system/konnectivity-agent-k2fjm" Oct 30 12:45:02.652763 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652498 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-snapshots\") pod \"insights-operator-7c4986d449-sztw6\" (UID: \"73b3e3a7-93fa-49c2-a8f1-4188a22b99a5\") " pod="openshift-insights/insights-operator-7c4986d449-sztw6" Oct 30 12:45:02.652763 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652506 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-multus-socket-dir-parent\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.652763 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652533 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-image-registry-private-configuration\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:45:02.652763 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652563 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bc5eaee0-b01e-432b-86ae-0d97a04da650-ovnkube-config\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.652763 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652588 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-tmp\") pod \"insights-operator-7c4986d449-sztw6\" (UID: \"73b3e3a7-93fa-49c2-a8f1-4188a22b99a5\") " pod="openshift-insights/insights-operator-7c4986d449-sztw6" Oct 30 12:45:02.652763 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652613 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-var-lib-kubelet\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.652763 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652637 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21ea018f-70e6-4ee3-952d-704089532abb-original-pull-secret\") pod \"global-pull-secret-syncer-nk9fb\" (UID: \"21ea018f-70e6-4ee3-952d-704089532abb\") " pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:45:02.652763 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652686 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/40168acf-dcfa-4323-91eb-e710a2b82caa-crio-socket\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:45:02.652763 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652688 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-tmp\") pod \"insights-operator-7c4986d449-sztw6\" (UID: \"73b3e3a7-93fa-49c2-a8f1-4188a22b99a5\") " pod="openshift-insights/insights-operator-7c4986d449-sztw6" Oct 30 12:45:02.652763 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652737 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/40168acf-dcfa-4323-91eb-e710a2b82caa-data-volume\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:45:02.652763 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652770 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcpmg\" (UniqueName: \"kubernetes.io/projected/e4bf0a41-a4bd-47ad-93b1-088500499acb-kube-api-access-lcpmg\") pod \"service-ca-operator-544d749c8c-kzhq7\" (UID: \"e4bf0a41-a4bd-47ad-93b1-088500499acb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzhq7" Oct 30 12:45:02.653303 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652786 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-sztw6\" (UID: \"73b3e3a7-93fa-49c2-a8f1-4188a22b99a5\") " pod="openshift-insights/insights-operator-7c4986d449-sztw6" Oct 30 12:45:02.653303 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652770 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-image-registry-private-configuration\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:45:02.653303 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652822 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/40168acf-dcfa-4323-91eb-e710a2b82caa-data-volume\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:45:02.653303 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652872 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zks6k\" (UniqueName: \"kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-kube-api-access-zks6k\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:45:02.653303 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652886 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bc5eaee0-b01e-432b-86ae-0d97a04da650-ovnkube-config\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.653303 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652891 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a6ee59d-3b62-4be1-87e8-79bb3bece069-cni-binary-copy\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:45:02.653303 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652917 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs4g4\" (UniqueName: \"kubernetes.io/projected/1a6ee59d-3b62-4be1-87e8-79bb3bece069-kube-api-access-gs4g4\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:45:02.653303 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652922 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21ea018f-70e6-4ee3-952d-704089532abb-original-pull-secret\") pod \"global-pull-secret-syncer-nk9fb\" (UID: \"21ea018f-70e6-4ee3-952d-704089532abb\") " pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:45:02.653303 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652935 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwpj2\" (UniqueName: \"kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-kube-api-access-jwpj2\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:45:02.653303 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652959 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-run-netns\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.653303 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652976 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-run-k8s-cni-cncf-io\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.653303 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.652991 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/877cfa0b-b1c6-4455-b232-b1ca558b5084-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5bmsj\" (UID: \"877cfa0b-b1c6-4455-b232-b1ca558b5084\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5bmsj" Oct 30 12:45:02.653303 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.653014 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-cni-bin\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.653303 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.653058 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-etc-kubernetes\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.653303 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.653079 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a6ee59d-3b62-4be1-87e8-79bb3bece069-cni-binary-copy\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:45:02.653303 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.653107 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-image-registry-private-configuration\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:45:02.653303 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.653232 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-ca-trust-extracted\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:45:02.653303 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.653308 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-ca-trust-extracted\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:45:02.653907 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.653319 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-installation-pull-secrets\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:45:02.653907 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.653346 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/49c39196-4e77-4bcf-86b1-705a263e394b-serviceca\") pod \"node-ca-rjhg6\" (UID: \"49c39196-4e77-4bcf-86b1-705a263e394b\") " pod="openshift-image-registry/node-ca-rjhg6" Oct 30 12:45:02.653907 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.653321 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-image-registry-private-configuration\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:45:02.653907 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.653362 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpcwv\" (UniqueName: \"kubernetes.io/projected/8107811a-ec7c-497a-83fc-372249074496-kube-api-access-jpcwv\") pod \"network-check-source-6f78d59f99-pkpds\" (UID: \"8107811a-ec7c-497a-83fc-372249074496\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-pkpds" Oct 30 12:45:02.653907 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.653393 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-node-log\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.653907 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.653412 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbwtv\" (UniqueName: \"kubernetes.io/projected/9c377915-c344-4791-baf6-4b286b9c019b-kube-api-access-mbwtv\") pod \"kas-connection-checker-685895c7c9-6xnwh\" (UID: \"9c377915-c344-4791-baf6-4b286b9c019b\") " pod="kube-system/kas-connection-checker-685895c7c9-6xnwh" Oct 30 12:45:02.653907 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.653429 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-host\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.653907 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.653452 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g248\" (UniqueName: \"kubernetes.io/projected/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-kube-api-access-4g248\") pod \"cluster-samples-operator-6ddb6c8b94-s7c7s\" (UID: \"4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-s7c7s" Oct 30 12:45:02.653907 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.653475 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-device-dir\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:45:02.653907 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.653490 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21ea018f-70e6-4ee3-952d-704089532abb-dbus\") pod \"global-pull-secret-syncer-nk9fb\" (UID: \"21ea018f-70e6-4ee3-952d-704089532abb\") " pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:45:02.653907 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.653513 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpmfr\" (UniqueName: \"kubernetes.io/projected/24a00317-8fb9-41f3-8e1a-d1d985887725-kube-api-access-lpmfr\") pod \"node-resolver-gt8nx\" (UID: \"24a00317-8fb9-41f3-8e1a-d1d985887725\") " pod="openshift-dns/node-resolver-gt8nx" Oct 30 12:45:02.653907 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.653553 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-var-lib-kubelet\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.653907 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.653577 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-installation-pull-secrets\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:45:02.653907 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.653543 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/49c39196-4e77-4bcf-86b1-705a263e394b-serviceca\") pod \"node-ca-rjhg6\" (UID: \"49c39196-4e77-4bcf-86b1-705a263e394b\") " pod="openshift-image-registry/node-ca-rjhg6" Oct 30 12:45:02.653907 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.653660 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49c39196-4e77-4bcf-86b1-705a263e394b-host\") pod \"node-ca-rjhg6\" (UID: \"49c39196-4e77-4bcf-86b1-705a263e394b\") " pod="openshift-image-registry/node-ca-rjhg6" Oct 30 12:45:02.653907 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.653690 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j94kk\" (UniqueName: \"kubernetes.io/projected/40168acf-dcfa-4323-91eb-e710a2b82caa-kube-api-access-j94kk\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:45:02.653907 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.653711 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24a00317-8fb9-41f3-8e1a-d1d985887725-tmp-dir\") pod \"node-resolver-gt8nx\" (UID: \"24a00317-8fb9-41f3-8e1a-d1d985887725\") " pod="openshift-dns/node-resolver-gt8nx" Oct 30 12:45:02.653907 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.653734 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-sysctl-conf\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.653907 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.653767 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-sys\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.653907 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.653788 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24a00317-8fb9-41f3-8e1a-d1d985887725-tmp-dir\") pod \"node-resolver-gt8nx\" (UID: \"24a00317-8fb9-41f3-8e1a-d1d985887725\") " pod="openshift-dns/node-resolver-gt8nx" Oct 30 12:45:02.653907 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.653815 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-default-certificate\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:45:02.653907 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.653832 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-sys-fs\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:45:02.653907 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.653860 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-log-socket\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.653907 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.653900 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wppl9\" (UniqueName: \"kubernetes.io/projected/64bdbc16-e845-45da-8f34-f0d788a75eac-kube-api-access-wppl9\") pod \"cluster-monitoring-operator-884576c57-vjdmx\" (UID: \"64bdbc16-e845-45da-8f34-f0d788a75eac\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjdmx" Oct 30 12:45:02.654828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.653933 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a050566-8e52-4ea5-a4b6-575b1b4c9548-cni-binary-copy\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.654828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.654029 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-hostroot\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.654828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.654048 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-default-certificate\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:45:02.654828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.654138 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8wc5\" (UniqueName: \"kubernetes.io/projected/93e1032c-06a8-4418-99b7-540d1a4f1232-kube-api-access-p8wc5\") pod \"volume-data-source-validator-58c9f5f46d-c8pgj\" (UID: \"93e1032c-06a8-4418-99b7-540d1a4f1232\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-c8pgj" Oct 30 12:45:02.654828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.654172 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/71e82393-029b-4388-86df-8bd068720867-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pptmg\" (UID: \"71e82393-029b-4388-86df-8bd068720867\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pptmg" Oct 30 12:45:02.654828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.654206 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a050566-8e52-4ea5-a4b6-575b1b4c9548-cni-binary-copy\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.654828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.654281 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7664727a-bcd2-4169-b70b-a0ecba045321-metrics-certs\") pod \"network-metrics-daemon-j6b2w\" (UID: \"7664727a-bcd2-4169-b70b-a0ecba045321\") " pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:45:02.654828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.654370 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0-iptables-alerter-script\") pod \"iptables-alerter-9tp7b\" (UID: \"1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0\") " pod="openshift-network-operator/iptables-alerter-9tp7b" Oct 30 12:45:02.654828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.654378 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/71e82393-029b-4388-86df-8bd068720867-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pptmg\" (UID: \"71e82393-029b-4388-86df-8bd068720867\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pptmg" Oct 30 12:45:02.654828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.654440 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7cb39685-b929-48bd-afd2-228862e62668-config-volume\") pod \"dns-default-w52p8\" (UID: \"7cb39685-b929-48bd-afd2-228862e62668\") " pod="openshift-dns/dns-default-w52p8" Oct 30 12:45:02.654828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.654495 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/64bdbc16-e845-45da-8f34-f0d788a75eac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vjdmx\" (UID: \"64bdbc16-e845-45da-8f34-f0d788a75eac\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjdmx" Oct 30 12:45:02.654828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.654544 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21ea018f-70e6-4ee3-952d-704089532abb-kubelet-config\") pod \"global-pull-secret-syncer-nk9fb\" (UID: \"21ea018f-70e6-4ee3-952d-704089532abb\") " pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:45:02.654828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.654571 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bc5eaee0-b01e-432b-86ae-0d97a04da650-ovnkube-script-lib\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.654828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.654582 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0-iptables-alerter-script\") pod \"iptables-alerter-9tp7b\" (UID: \"1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0\") " pod="openshift-network-operator/iptables-alerter-9tp7b" Oct 30 12:45:02.654828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.654590 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40168acf-dcfa-4323-91eb-e710a2b82caa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:45:02.654828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.654624 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/24a00317-8fb9-41f3-8e1a-d1d985887725-hosts-file\") pod \"node-resolver-gt8nx\" (UID: \"24a00317-8fb9-41f3-8e1a-d1d985887725\") " pod="openshift-dns/node-resolver-gt8nx" Oct 30 12:45:02.654828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.654651 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7cb39685-b929-48bd-afd2-228862e62668-config-volume\") pod \"dns-default-w52p8\" (UID: \"7cb39685-b929-48bd-afd2-228862e62668\") " pod="openshift-dns/dns-default-w52p8" Oct 30 12:45:02.654828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.654665 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-modprobe-d\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.654828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.654711 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-kubernetes\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.654828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.654741 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-tuned\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.654828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.654747 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bc5eaee0-b01e-432b-86ae-0d97a04da650-ovnkube-script-lib\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.654828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.654757 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2bdf343-b9cd-484a-a8c1-1d0713d74093-tmp\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.654828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.654776 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-sysconfig\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.654828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.654807 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2bdf343-b9cd-484a-a8c1-1d0713d74093-tmp\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.654828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.654816 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-tuned\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.654828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.654825 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6xkw\" (UniqueName: \"kubernetes.io/projected/7cb39685-b929-48bd-afd2-228862e62668-kube-api-access-p6xkw\") pod \"dns-default-w52p8\" (UID: \"7cb39685-b929-48bd-afd2-228862e62668\") " pod="openshift-dns/dns-default-w52p8" Oct 30 12:45:02.655690 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.654866 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-var-lib-cni-bin\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.655690 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.654886 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-multus-conf-dir\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.655690 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.654904 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpzms\" (UniqueName: \"kubernetes.io/projected/1d06e291-b9d3-48f7-8fc2-f615440631d9-kube-api-access-jpzms\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:45:02.655690 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.654933 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:45:02.655690 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.655021 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-trusted-ca\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:45:02.655690 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.655041 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-bound-sa-token\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:45:02.655690 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.655082 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-installation-pull-secrets\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:45:02.655690 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.655195 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40168acf-dcfa-4323-91eb-e710a2b82caa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:45:02.655690 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.655304 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-systemd\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.655690 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.655348 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8a050566-8e52-4ea5-a4b6-575b1b4c9548-multus-daemon-config\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.655690 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.655355 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-installation-pull-secrets\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:45:02.655690 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.655361 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-trusted-ca\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:45:02.655690 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.655382 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-trusted-ca\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:45:02.655690 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.655512 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8a050566-8e52-4ea5-a4b6-575b1b4c9548-multus-daemon-config\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.655690 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.655518 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-slash\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.655690 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.655603 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-run-ovn-kubernetes\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.655690 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.655635 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6rnd\" (UniqueName: \"kubernetes.io/projected/8a050566-8e52-4ea5-a4b6-575b1b4c9548-kube-api-access-v6rnd\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.655690 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.655653 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-s7c7s\" (UID: \"4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-s7c7s" Oct 30 12:45:02.656470 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.655701 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-certificates\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:45:02.656470 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.655793 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-s7c7s\" (UID: \"4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-s7c7s" Oct 30 12:45:02.656470 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.655827 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbk56\" (UniqueName: \"kubernetes.io/projected/b2bdf343-b9cd-484a-a8c1-1d0713d74093-kube-api-access-cbk56\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.656470 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.655862 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4df70e16-5acb-494e-ab67-855c809e3411-cert\") pod \"ingress-canary-dnpgs\" (UID: \"4df70e16-5acb-494e-ab67-855c809e3411\") " pod="openshift-ingress-canary/ingress-canary-dnpgs" Oct 30 12:45:02.656470 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.655880 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-certificates\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:45:02.656470 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.655935 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-run-systemd\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.656470 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.655982 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e4bf0a41-a4bd-47ad-93b1-088500499acb-serving-cert\") pod \"service-ca-operator-544d749c8c-kzhq7\" (UID: \"e4bf0a41-a4bd-47ad-93b1-088500499acb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzhq7" Oct 30 12:45:02.656470 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.656012 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-bound-sa-token\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:45:02.656470 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.656046 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-run-ovn\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.656470 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.656067 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bc5eaee0-b01e-432b-86ae-0d97a04da650-env-overrides\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.656470 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.656082 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cb39685-b929-48bd-afd2-228862e62668-metrics-tls\") pod \"dns-default-w52p8\" (UID: \"7cb39685-b929-48bd-afd2-228862e62668\") " pod="openshift-dns/dns-default-w52p8" Oct 30 12:45:02.656470 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.656121 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4df70e16-5acb-494e-ab67-855c809e3411-cert\") pod \"ingress-canary-dnpgs\" (UID: \"4df70e16-5acb-494e-ab67-855c809e3411\") " pod="openshift-ingress-canary/ingress-canary-dnpgs" Oct 30 12:45:02.656470 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.656166 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e4bf0a41-a4bd-47ad-93b1-088500499acb-serving-cert\") pod \"service-ca-operator-544d749c8c-kzhq7\" (UID: \"e4bf0a41-a4bd-47ad-93b1-088500499acb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzhq7" Oct 30 12:45:02.656470 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.656181 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/71e82393-029b-4388-86df-8bd068720867-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pptmg\" (UID: \"71e82393-029b-4388-86df-8bd068720867\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pptmg" Oct 30 12:45:02.656470 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.656216 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-trusted-ca\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:45:02.656470 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.656233 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-socket-dir\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:45:02.656470 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.656302 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-kubelet\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.656470 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.656309 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bc5eaee0-b01e-432b-86ae-0d97a04da650-env-overrides\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.656470 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.656329 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnz9b\" (UniqueName: \"kubernetes.io/projected/ae9fe0c2-0d00-4891-8549-bc99f5373da9-kube-api-access-pnz9b\") pod \"kas-connection-checker-685895c7c9-cv78r\" (UID: \"ae9fe0c2-0d00-4891-8549-bc99f5373da9\") " pod="kube-system/kas-connection-checker-685895c7c9-cv78r" Oct 30 12:45:02.656470 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.656414 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/71e82393-029b-4388-86df-8bd068720867-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pptmg\" (UID: \"71e82393-029b-4388-86df-8bd068720867\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pptmg" Oct 30 12:45:02.656470 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.656431 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-multus-cni-dir\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.657349 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.656487 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d06e291-b9d3-48f7-8fc2-f615440631d9-service-ca-bundle\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:45:02.657349 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.656510 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/37620bdd-11a4-4c12-8be1-9813a85d575b-konnectivity-ca\") pod \"konnectivity-agent-k2fjm\" (UID: \"37620bdd-11a4-4c12-8be1-9813a85d575b\") " pod="kube-system/konnectivity-agent-k2fjm" Oct 30 12:45:02.657349 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.656527 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-service-ca-bundle\") pod \"insights-operator-7c4986d449-sztw6\" (UID: \"73b3e3a7-93fa-49c2-a8f1-4188a22b99a5\") " pod="openshift-insights/insights-operator-7c4986d449-sztw6" Oct 30 12:45:02.657349 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.656625 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-cnibin\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.657349 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.656676 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzptg\" (UniqueName: \"kubernetes.io/projected/7664727a-bcd2-4169-b70b-a0ecba045321-kube-api-access-vzptg\") pod \"network-metrics-daemon-j6b2w\" (UID: \"7664727a-bcd2-4169-b70b-a0ecba045321\") " pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:45:02.657349 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.656686 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/37620bdd-11a4-4c12-8be1-9813a85d575b-konnectivity-ca\") pod \"konnectivity-agent-k2fjm\" (UID: \"37620bdd-11a4-4c12-8be1-9813a85d575b\") " pod="kube-system/konnectivity-agent-k2fjm" Oct 30 12:45:02.657349 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.656705 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-certificates\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:45:02.657349 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.656730 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2szrp\" (UniqueName: \"kubernetes.io/projected/1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0-kube-api-access-2szrp\") pod \"iptables-alerter-9tp7b\" (UID: \"1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0\") " pod="openshift-network-operator/iptables-alerter-9tp7b" Oct 30 12:45:02.657349 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.656762 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bc5eaee0-b01e-432b-86ae-0d97a04da650-ovn-node-metrics-cert\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.657349 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.656801 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-service-ca-bundle\") pod \"insights-operator-7c4986d449-sztw6\" (UID: \"73b3e3a7-93fa-49c2-a8f1-4188a22b99a5\") " pod="openshift-insights/insights-operator-7c4986d449-sztw6" Oct 30 12:45:02.657349 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.656804 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-metrics-certs\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:45:02.657349 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.656839 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-certificates\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:45:02.657349 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.656853 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-etc-selinux\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:45:02.657349 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.656882 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/877cfa0b-b1c6-4455-b232-b1ca558b5084-nginx-conf\") pod \"networking-console-plugin-86c478bb88-5bmsj\" (UID: \"877cfa0b-b1c6-4455-b232-b1ca558b5084\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5bmsj" Oct 30 12:45:02.657349 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.656909 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00ee6aae-7264-4c8d-a337-c7c9f1132e9c-trusted-ca\") pod \"console-operator-7cd56b8dbc-v4mks\" (UID: \"00ee6aae-7264-4c8d-a337-c7c9f1132e9c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" Oct 30 12:45:02.657349 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.656934 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-tls\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:45:02.657349 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.656961 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bc5eaee0-b01e-432b-86ae-0d97a04da650-ovn-node-metrics-cert\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.657349 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.656962 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-ca-trust-extracted\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:45:02.657349 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.657003 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-registration-dir\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:45:02.657349 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.657010 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/877cfa0b-b1c6-4455-b232-b1ca558b5084-nginx-conf\") pod \"networking-console-plugin-86c478bb88-5bmsj\" (UID: \"877cfa0b-b1c6-4455-b232-b1ca558b5084\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5bmsj" Oct 30 12:45:02.657349 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.657033 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h92pg\" (UniqueName: \"kubernetes.io/projected/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-kube-api-access-h92pg\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:45:02.657349 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.657030 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-ca-trust-extracted\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:45:02.657349 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.657061 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-tls\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:45:02.657349 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.657083 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdkm8\" (UniqueName: \"kubernetes.io/projected/b32002db-049b-4456-91d7-66c33a85c040-kube-api-access-pdkm8\") pod \"migrator-66df5d496b-4h8hg\" (UID: \"b32002db-049b-4456-91d7-66c33a85c040\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4h8hg" Oct 30 12:45:02.657349 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.657122 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn2wd\" (UniqueName: \"kubernetes.io/projected/49c39196-4e77-4bcf-86b1-705a263e394b-kube-api-access-sn2wd\") pod \"node-ca-rjhg6\" (UID: \"49c39196-4e77-4bcf-86b1-705a263e394b\") " pod="openshift-image-registry/node-ca-rjhg6" Oct 30 12:45:02.657349 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.657227 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00ee6aae-7264-4c8d-a337-c7c9f1132e9c-trusted-ca\") pod \"console-operator-7cd56b8dbc-v4mks\" (UID: \"00ee6aae-7264-4c8d-a337-c7c9f1132e9c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" Oct 30 12:45:02.657349 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.657251 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1a6ee59d-3b62-4be1-87e8-79bb3bece069-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:45:02.657349 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.657303 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dkqx\" (UniqueName: \"kubernetes.io/projected/bc5eaee0-b01e-432b-86ae-0d97a04da650-kube-api-access-5dkqx\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.657349 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.657331 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-sysctl-d\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.657349 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.657369 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a6ee59d-3b62-4be1-87e8-79bb3bece069-system-cni-dir\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:45:02.658228 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.657394 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-run-openvswitch\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.658228 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.657417 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-run\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.658228 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.657439 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1a6ee59d-3b62-4be1-87e8-79bb3bece069-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:45:02.658228 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.657455 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/64bdbc16-e845-45da-8f34-f0d788a75eac-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-vjdmx\" (UID: \"64bdbc16-e845-45da-8f34-f0d788a75eac\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjdmx" Oct 30 12:45:02.658228 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.657535 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-run-netns\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.658228 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.657566 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a6ee59d-3b62-4be1-87e8-79bb3bece069-cnibin\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:45:02.658228 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.657612 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.658228 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.657657 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cb39685-b929-48bd-afd2-228862e62668-tmp-dir\") pod \"dns-default-w52p8\" (UID: \"7cb39685-b929-48bd-afd2-228862e62668\") " pod="openshift-dns/dns-default-w52p8" Oct 30 12:45:02.658228 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.657703 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-s7c7s\" (UID: \"4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-s7c7s" Oct 30 12:45:02.658228 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.657717 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/64bdbc16-e845-45da-8f34-f0d788a75eac-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-vjdmx\" (UID: \"64bdbc16-e845-45da-8f34-f0d788a75eac\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjdmx" Oct 30 12:45:02.658228 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.657749 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cb39685-b929-48bd-afd2-228862e62668-tmp-dir\") pod \"dns-default-w52p8\" (UID: \"7cb39685-b929-48bd-afd2-228862e62668\") " pod="openshift-dns/dns-default-w52p8" Oct 30 12:45:02.658228 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.657778 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-stats-auth\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:45:02.658228 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.657888 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkdqk\" (UniqueName: \"kubernetes.io/projected/4df70e16-5acb-494e-ab67-855c809e3411-kube-api-access-hkdqk\") pod \"ingress-canary-dnpgs\" (UID: \"4df70e16-5acb-494e-ab67-855c809e3411\") " pod="openshift-ingress-canary/ingress-canary-dnpgs" Oct 30 12:45:02.658228 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.657916 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-var-lib-openvswitch\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.658228 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.657933 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-cni-netd\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.658228 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.657939 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-stats-auth\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:45:02.677328 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.677303 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtcpz\" (UniqueName: \"kubernetes.io/projected/00ee6aae-7264-4c8d-a337-c7c9f1132e9c-kube-api-access-wtcpz\") pod \"console-operator-7cd56b8dbc-v4mks\" (UID: \"00ee6aae-7264-4c8d-a337-c7c9f1132e9c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" Oct 30 12:45:02.677705 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.677688 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf8zf\" (UniqueName: \"kubernetes.io/projected/a3928331-ce67-4c37-9c5d-63b0da642a0f-kube-api-access-vf8zf\") pod \"kas-connection-checker-685895c7c9-rnflh\" (UID: \"a3928331-ce67-4c37-9c5d-63b0da642a0f\") " pod="kube-system/kas-connection-checker-685895c7c9-rnflh" Oct 30 12:45:02.677779 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.677764 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4rqz\" (UniqueName: \"kubernetes.io/projected/eb4ec7a3-2fd7-44dd-98ca-72710972a050-kube-api-access-j4rqz\") pod \"network-check-target-ljvqz\" (UID: \"eb4ec7a3-2fd7-44dd-98ca-72710972a050\") " pod="openshift-network-diagnostics/network-check-target-ljvqz" Oct 30 12:45:02.677934 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.677912 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbnzh\" (UniqueName: \"kubernetes.io/projected/71e82393-029b-4388-86df-8bd068720867-kube-api-access-kbnzh\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pptmg\" (UID: \"71e82393-029b-4388-86df-8bd068720867\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pptmg" Oct 30 12:45:02.678107 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.678087 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfff9\" (UniqueName: \"kubernetes.io/projected/73b3e3a7-93fa-49c2-a8f1-4188a22b99a5-kube-api-access-wfff9\") pod \"insights-operator-7c4986d449-sztw6\" (UID: \"73b3e3a7-93fa-49c2-a8f1-4188a22b99a5\") " pod="openshift-insights/insights-operator-7c4986d449-sztw6" Oct 30 12:45:02.681185 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.681168 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcpmg\" (UniqueName: \"kubernetes.io/projected/e4bf0a41-a4bd-47ad-93b1-088500499acb-kube-api-access-lcpmg\") pod \"service-ca-operator-544d749c8c-kzhq7\" (UID: \"e4bf0a41-a4bd-47ad-93b1-088500499acb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzhq7" Oct 30 12:45:02.684606 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.684590 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn2wd\" (UniqueName: \"kubernetes.io/projected/49c39196-4e77-4bcf-86b1-705a263e394b-kube-api-access-sn2wd\") pod \"node-ca-rjhg6\" (UID: \"49c39196-4e77-4bcf-86b1-705a263e394b\") " pod="openshift-image-registry/node-ca-rjhg6" Oct 30 12:45:02.684671 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.684611 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6xkw\" (UniqueName: \"kubernetes.io/projected/7cb39685-b929-48bd-afd2-228862e62668-kube-api-access-p6xkw\") pod \"dns-default-w52p8\" (UID: \"7cb39685-b929-48bd-afd2-228862e62668\") " pod="openshift-dns/dns-default-w52p8" Oct 30 12:45:02.687015 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.686997 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkdqk\" (UniqueName: \"kubernetes.io/projected/4df70e16-5acb-494e-ab67-855c809e3411-kube-api-access-hkdqk\") pod \"ingress-canary-dnpgs\" (UID: \"4df70e16-5acb-494e-ab67-855c809e3411\") " pod="openshift-ingress-canary/ingress-canary-dnpgs" Oct 30 12:45:02.688373 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.688356 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2szrp\" (UniqueName: \"kubernetes.io/projected/1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0-kube-api-access-2szrp\") pod \"iptables-alerter-9tp7b\" (UID: \"1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0\") " pod="openshift-network-operator/iptables-alerter-9tp7b" Oct 30 12:45:02.688631 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.688616 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-bound-sa-token\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:45:02.688752 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.688727 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h92pg\" (UniqueName: \"kubernetes.io/projected/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-kube-api-access-h92pg\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:45:02.689071 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.688598 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzptg\" (UniqueName: \"kubernetes.io/projected/7664727a-bcd2-4169-b70b-a0ecba045321-kube-api-access-vzptg\") pod \"network-metrics-daemon-j6b2w\" (UID: \"7664727a-bcd2-4169-b70b-a0ecba045321\") " pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:45:02.689831 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.689785 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnz9b\" (UniqueName: \"kubernetes.io/projected/ae9fe0c2-0d00-4891-8549-bc99f5373da9-kube-api-access-pnz9b\") pod \"kas-connection-checker-685895c7c9-cv78r\" (UID: \"ae9fe0c2-0d00-4891-8549-bc99f5373da9\") " pod="kube-system/kas-connection-checker-685895c7c9-cv78r" Oct 30 12:45:02.689898 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.689880 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6rnd\" (UniqueName: \"kubernetes.io/projected/8a050566-8e52-4ea5-a4b6-575b1b4c9548-kube-api-access-v6rnd\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.690606 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.690588 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwpj2\" (UniqueName: \"kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-kube-api-access-jwpj2\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:45:02.692927 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.692910 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs4g4\" (UniqueName: \"kubernetes.io/projected/1a6ee59d-3b62-4be1-87e8-79bb3bece069-kube-api-access-gs4g4\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:45:02.693441 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.693422 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdkm8\" (UniqueName: \"kubernetes.io/projected/b32002db-049b-4456-91d7-66c33a85c040-kube-api-access-pdkm8\") pod \"migrator-66df5d496b-4h8hg\" (UID: \"b32002db-049b-4456-91d7-66c33a85c040\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4h8hg" Oct 30 12:45:02.693595 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.693579 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpzms\" (UniqueName: \"kubernetes.io/projected/1d06e291-b9d3-48f7-8fc2-f615440631d9-kube-api-access-jpzms\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:45:02.693821 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.693802 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbwtv\" (UniqueName: \"kubernetes.io/projected/9c377915-c344-4791-baf6-4b286b9c019b-kube-api-access-mbwtv\") pod \"kas-connection-checker-685895c7c9-6xnwh\" (UID: \"9c377915-c344-4791-baf6-4b286b9c019b\") " pod="kube-system/kas-connection-checker-685895c7c9-6xnwh" Oct 30 12:45:02.694102 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.694084 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpcwv\" (UniqueName: \"kubernetes.io/projected/8107811a-ec7c-497a-83fc-372249074496-kube-api-access-jpcwv\") pod \"network-check-source-6f78d59f99-pkpds\" (UID: \"8107811a-ec7c-497a-83fc-372249074496\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-pkpds" Oct 30 12:45:02.694167 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.694134 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dkqx\" (UniqueName: \"kubernetes.io/projected/bc5eaee0-b01e-432b-86ae-0d97a04da650-kube-api-access-5dkqx\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.694243 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.694220 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbk56\" (UniqueName: \"kubernetes.io/projected/b2bdf343-b9cd-484a-a8c1-1d0713d74093-kube-api-access-cbk56\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.694380 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.694357 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8wc5\" (UniqueName: \"kubernetes.io/projected/93e1032c-06a8-4418-99b7-540d1a4f1232-kube-api-access-p8wc5\") pod \"volume-data-source-validator-58c9f5f46d-c8pgj\" (UID: \"93e1032c-06a8-4418-99b7-540d1a4f1232\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-c8pgj" Oct 30 12:45:02.694858 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.694842 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g248\" (UniqueName: \"kubernetes.io/projected/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-kube-api-access-4g248\") pod \"cluster-samples-operator-6ddb6c8b94-s7c7s\" (UID: \"4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-s7c7s" Oct 30 12:45:02.695579 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.695563 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpmfr\" (UniqueName: \"kubernetes.io/projected/24a00317-8fb9-41f3-8e1a-d1d985887725-kube-api-access-lpmfr\") pod \"node-resolver-gt8nx\" (UID: \"24a00317-8fb9-41f3-8e1a-d1d985887725\") " pod="openshift-dns/node-resolver-gt8nx" Oct 30 12:45:02.695828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.695813 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wppl9\" (UniqueName: \"kubernetes.io/projected/64bdbc16-e845-45da-8f34-f0d788a75eac-kube-api-access-wppl9\") pod \"cluster-monitoring-operator-884576c57-vjdmx\" (UID: \"64bdbc16-e845-45da-8f34-f0d788a75eac\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjdmx" Oct 30 12:45:02.697388 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.697366 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-bound-sa-token\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:45:02.698412 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.698396 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j94kk\" (UniqueName: \"kubernetes.io/projected/40168acf-dcfa-4323-91eb-e710a2b82caa-kube-api-access-j94kk\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:45:02.699307 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.699290 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zks6k\" (UniqueName: \"kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-kube-api-access-zks6k\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:45:02.758464 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.758410 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-var-lib-kubelet\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.758464 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.758435 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/40168acf-dcfa-4323-91eb-e710a2b82caa-crio-socket\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:45:02.758464 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.758451 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-run-netns\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.758699 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.758466 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-run-k8s-cni-cncf-io\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.758699 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.758483 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/877cfa0b-b1c6-4455-b232-b1ca558b5084-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5bmsj\" (UID: \"877cfa0b-b1c6-4455-b232-b1ca558b5084\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5bmsj" Oct 30 12:45:02.758699 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.758506 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-cni-bin\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.758699 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.758522 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/40168acf-dcfa-4323-91eb-e710a2b82caa-crio-socket\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:45:02.758699 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.758530 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-etc-kubernetes\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.758699 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.758538 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-run-netns\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.758699 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.758525 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-var-lib-kubelet\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.758699 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.758553 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-run-k8s-cni-cncf-io\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.758699 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.758573 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-node-log\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.758699 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.758592 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-host\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.758699 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.758597 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-etc-kubernetes\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.758699 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.758561 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-cni-bin\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.758699 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.758627 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-node-log\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.758699 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.758606 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-device-dir\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:45:02.758699 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.758647 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-host\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.758699 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.758656 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21ea018f-70e6-4ee3-952d-704089532abb-dbus\") pod \"global-pull-secret-syncer-nk9fb\" (UID: \"21ea018f-70e6-4ee3-952d-704089532abb\") " pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:45:02.758699 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.758674 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-var-lib-kubelet\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.758699 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.758674 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-device-dir\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:45:02.758699 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:02.758685 4931 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 12:45:02.758699 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.758710 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49c39196-4e77-4bcf-86b1-705a263e394b-host\") pod \"node-ca-rjhg6\" (UID: \"49c39196-4e77-4bcf-86b1-705a263e394b\") " pod="openshift-image-registry/node-ca-rjhg6" Oct 30 12:45:02.759497 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.758720 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21ea018f-70e6-4ee3-952d-704089532abb-dbus\") pod \"global-pull-secret-syncer-nk9fb\" (UID: \"21ea018f-70e6-4ee3-952d-704089532abb\") " pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:45:02.759497 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.758729 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-var-lib-kubelet\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.759497 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:02.758771 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/877cfa0b-b1c6-4455-b232-b1ca558b5084-networking-console-plugin-cert podName:877cfa0b-b1c6-4455-b232-b1ca558b5084 nodeName:}" failed. No retries permitted until 2026-10-30 12:45:03.258756412 +0000 UTC m=+1.862254836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/877cfa0b-b1c6-4455-b232-b1ca558b5084-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5bmsj" (UID: "877cfa0b-b1c6-4455-b232-b1ca558b5084") : secret "networking-console-plugin-cert" not found Oct 30 12:45:02.759497 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.758688 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49c39196-4e77-4bcf-86b1-705a263e394b-host\") pod \"node-ca-rjhg6\" (UID: \"49c39196-4e77-4bcf-86b1-705a263e394b\") " pod="openshift-image-registry/node-ca-rjhg6" Oct 30 12:45:02.759497 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.758805 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-sysctl-conf\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.759497 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.758820 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-sys\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.759497 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.758836 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-sys-fs\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:45:02.759497 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.758850 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-log-socket\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.759497 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.758875 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-hostroot\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.759497 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.758886 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-sys\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.759497 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.759203 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-hostroot\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.759497 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.758896 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7664727a-bcd2-4169-b70b-a0ecba045321-metrics-certs\") pod \"network-metrics-daemon-j6b2w\" (UID: \"7664727a-bcd2-4169-b70b-a0ecba045321\") " pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:45:02.759497 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.759215 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-log-socket\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.759497 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.759202 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-sys-fs\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:45:02.759497 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.759242 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-sysctl-conf\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.759497 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.759309 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/64bdbc16-e845-45da-8f34-f0d788a75eac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vjdmx\" (UID: \"64bdbc16-e845-45da-8f34-f0d788a75eac\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjdmx" Oct 30 12:45:02.759497 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.759381 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21ea018f-70e6-4ee3-952d-704089532abb-kubelet-config\") pod \"global-pull-secret-syncer-nk9fb\" (UID: \"21ea018f-70e6-4ee3-952d-704089532abb\") " pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:45:02.759497 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:02.759390 4931 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 12:45:02.759497 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.759416 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40168acf-dcfa-4323-91eb-e710a2b82caa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:45:02.759497 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:02.759456 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64bdbc16-e845-45da-8f34-f0d788a75eac-cluster-monitoring-operator-tls podName:64bdbc16-e845-45da-8f34-f0d788a75eac nodeName:}" failed. No retries permitted until 2026-10-30 12:45:03.259430747 +0000 UTC m=+1.862929173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/64bdbc16-e845-45da-8f34-f0d788a75eac-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vjdmx" (UID: "64bdbc16-e845-45da-8f34-f0d788a75eac") : secret "cluster-monitoring-operator-tls" not found Oct 30 12:45:02.760378 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:02.759582 4931 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 12:45:02.760378 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:02.759644 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7664727a-bcd2-4169-b70b-a0ecba045321-metrics-certs podName:7664727a-bcd2-4169-b70b-a0ecba045321 nodeName:}" failed. No retries permitted until 2026-10-30 12:45:03.25963187 +0000 UTC m=+1.863130320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7664727a-bcd2-4169-b70b-a0ecba045321-metrics-certs") pod "network-metrics-daemon-j6b2w" (UID: "7664727a-bcd2-4169-b70b-a0ecba045321") : secret "metrics-daemon-secret" not found Oct 30 12:45:02.760378 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.759699 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21ea018f-70e6-4ee3-952d-704089532abb-kubelet-config\") pod \"global-pull-secret-syncer-nk9fb\" (UID: \"21ea018f-70e6-4ee3-952d-704089532abb\") " pod="kube-system/global-pull-secret-syncer-nk9fb" Oct 30 12:45:02.760378 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.759740 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/24a00317-8fb9-41f3-8e1a-d1d985887725-hosts-file\") pod \"node-resolver-gt8nx\" (UID: \"24a00317-8fb9-41f3-8e1a-d1d985887725\") " pod="openshift-dns/node-resolver-gt8nx" Oct 30 12:45:02.760378 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.759785 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-modprobe-d\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.760378 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.759816 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-kubernetes\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.760378 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.759852 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-sysconfig\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.760378 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.759885 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-var-lib-cni-bin\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.760378 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.759916 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-multus-conf-dir\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.760378 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.759946 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:45:02.760378 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.759960 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40168acf-dcfa-4323-91eb-e710a2b82caa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:45:02.760378 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.759976 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40168acf-dcfa-4323-91eb-e710a2b82caa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:45:02.760378 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.760007 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-systemd\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.760378 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.760038 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-slash\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.760378 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.760048 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-sysconfig\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.760378 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.760069 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-run-ovn-kubernetes\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.760378 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.760106 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-run-systemd\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.760378 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.760114 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/24a00317-8fb9-41f3-8e1a-d1d985887725-hosts-file\") pod \"node-resolver-gt8nx\" (UID: \"24a00317-8fb9-41f3-8e1a-d1d985887725\") " pod="openshift-dns/node-resolver-gt8nx" Oct 30 12:45:02.760378 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.760133 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-run-ovn\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.760378 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.760166 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cb39685-b929-48bd-afd2-228862e62668-metrics-tls\") pod \"dns-default-w52p8\" (UID: \"7cb39685-b929-48bd-afd2-228862e62668\") " pod="openshift-dns/dns-default-w52p8" Oct 30 12:45:02.760378 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.760199 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-socket-dir\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:45:02.760378 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.760214 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-modprobe-d\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.760378 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.760233 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-kubelet\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.760378 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.760275 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-multus-cni-dir\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.760378 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.760287 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-kubernetes\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.760378 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.760302 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d06e291-b9d3-48f7-8fc2-f615440631d9-service-ca-bundle\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:45:02.760378 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.760334 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-cnibin\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.760378 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.760340 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-run-ovn-kubernetes\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.760378 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.760369 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-metrics-certs\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:45:02.761544 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.760400 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-etc-selinux\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:45:02.761544 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:02.760554 4931 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 12:45:02.761544 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:02.760618 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40168acf-dcfa-4323-91eb-e710a2b82caa-insights-runtime-extractor-tls podName:40168acf-dcfa-4323-91eb-e710a2b82caa nodeName:}" failed. No retries permitted until 2026-10-30 12:45:03.260602675 +0000 UTC m=+1.864101113 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40168acf-dcfa-4323-91eb-e710a2b82caa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b94k6" (UID: "40168acf-dcfa-4323-91eb-e710a2b82caa") : secret "insights-runtime-extractor-tls" not found Oct 30 12:45:02.761544 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.760671 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-var-lib-cni-bin\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.761544 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.760729 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-systemd\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.761544 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.760770 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-multus-conf-dir\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.761544 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.760819 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:45:02.761544 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.760865 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-slash\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.761544 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.760906 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-kubelet\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.761544 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.760947 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-run-systemd\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.761544 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.761000 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-run-ovn\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.761544 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:02.761075 4931 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 12:45:02.761544 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:02.761119 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cb39685-b929-48bd-afd2-228862e62668-metrics-tls podName:7cb39685-b929-48bd-afd2-228862e62668 nodeName:}" failed. No retries permitted until 2026-10-30 12:45:03.2611057 +0000 UTC m=+1.864604138 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cb39685-b929-48bd-afd2-228862e62668-metrics-tls") pod "dns-default-w52p8" (UID: "7cb39685-b929-48bd-afd2-228862e62668") : secret "dns-default-metrics-tls" not found Oct 30 12:45:02.761544 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.761180 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-socket-dir\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:45:02.761544 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.761213 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-tls\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:45:02.761544 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.761248 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-registration-dir\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:45:02.761544 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.761299 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-tls\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:45:02.761544 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.761325 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-sysctl-d\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.761544 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.761358 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a6ee59d-3b62-4be1-87e8-79bb3bece069-system-cni-dir\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:45:02.761544 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.761387 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-run-openvswitch\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.761544 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.761418 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-run\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.761544 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.761441 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-run-netns\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.761544 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.761468 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a6ee59d-3b62-4be1-87e8-79bb3bece069-cnibin\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:45:02.761544 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.761499 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.761544 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.761533 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-s7c7s\" (UID: \"4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-s7c7s" Oct 30 12:45:02.762645 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.761564 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-var-lib-openvswitch\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.762645 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.761585 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-cni-netd\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.762645 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.761615 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0-host-slash\") pod \"iptables-alerter-9tp7b\" (UID: \"1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0\") " pod="openshift-network-operator/iptables-alerter-9tp7b" Oct 30 12:45:02.762645 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.761642 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-etc-openvswitch\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.762645 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.761669 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-system-cni-dir\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.762645 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.761697 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-var-lib-cni-multus\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.762645 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.761722 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-systemd-units\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.762645 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.761755 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a6ee59d-3b62-4be1-87e8-79bb3bece069-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:45:02.762645 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.761782 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-run-multus-certs\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.762645 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.761818 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-lib-modules\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.762645 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.761848 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-multus-socket-dir-parent\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.762645 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.761972 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-multus-socket-dir-parent\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.762645 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.762028 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-cnibin\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.762645 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.762085 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-multus-cni-dir\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.762645 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.762141 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-etc-selinux\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:45:02.762645 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:02.762235 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1d06e291-b9d3-48f7-8fc2-f615440631d9-service-ca-bundle podName:1d06e291-b9d3-48f7-8fc2-f615440631d9 nodeName:}" failed. No retries permitted until 2026-10-30 12:45:03.262222353 +0000 UTC m=+1.865720780 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1d06e291-b9d3-48f7-8fc2-f615440631d9-service-ca-bundle") pod "router-default-5db587c4bc-dttsw" (UID: "1d06e291-b9d3-48f7-8fc2-f615440631d9") : configmap references non-existent config key: service-ca.crt Oct 30 12:45:02.762645 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:02.762331 4931 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 12:45:02.762645 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:02.762342 4931 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bff985b5d-qwb5l: secret "image-registry-tls" not found Oct 30 12:45:02.762645 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:02.762407 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-tls podName:d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5 nodeName:}" failed. No retries permitted until 2026-10-30 12:45:03.262394882 +0000 UTC m=+1.865893324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-tls") pod "image-registry-7bff985b5d-qwb5l" (UID: "d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5") : secret "image-registry-tls" not found Oct 30 12:45:02.763476 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.762681 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4e40c7ec-9b38-4f24-8d7b-5689f029c0a3-registration-dir\") pod \"aws-ebs-csi-driver-node-7d82v\" (UID: \"4e40c7ec-9b38-4f24-8d7b-5689f029c0a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7d82v" Oct 30 12:45:02.763476 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.762753 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-var-lib-openvswitch\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.763476 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:02.762850 4931 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 12:45:02.763476 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:02.762862 4931 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-596f5cc4c5-sgvjw: secret "image-registry-tls" not found Oct 30 12:45:02.763476 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:02.762941 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-tls podName:6000e43a-e175-46fe-b5f7-a417fc1ba5bb nodeName:}" failed. No retries permitted until 2026-10-30 12:45:03.26292713 +0000 UTC m=+1.866425568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-tls") pod "image-registry-596f5cc4c5-sgvjw" (UID: "6000e43a-e175-46fe-b5f7-a417fc1ba5bb") : secret "image-registry-tls" not found Oct 30 12:45:02.763476 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.763060 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-etc-sysctl-d\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.763476 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.763106 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a6ee59d-3b62-4be1-87e8-79bb3bece069-system-cni-dir\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:45:02.763476 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.763158 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.763476 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.763244 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-system-cni-dir\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.763476 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.763237 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-run-netns\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.763476 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.763293 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a6ee59d-3b62-4be1-87e8-79bb3bece069-cnibin\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:45:02.763476 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.763316 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-run-multus-certs\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.763476 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.763349 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-run-openvswitch\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.763476 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:02.763380 4931 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 12:45:02.763476 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.763404 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8a050566-8e52-4ea5-a4b6-575b1b4c9548-host-var-lib-cni-multus\") pod \"multus-r54rk\" (UID: \"8a050566-8e52-4ea5-a4b6-575b1b4c9548\") " pod="openshift-multus/multus-r54rk" Oct 30 12:45:02.763476 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:02.763424 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-tls podName:4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c nodeName:}" failed. No retries permitted until 2026-10-30 12:45:03.263409459 +0000 UTC m=+1.866907883 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-s7c7s" (UID: "4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c") : secret "samples-operator-tls" not found Oct 30 12:45:02.763476 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.763450 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-etc-openvswitch\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.763476 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.763460 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-systemd-units\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.763476 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.763478 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-run\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.764128 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.763503 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bc5eaee0-b01e-432b-86ae-0d97a04da650-host-cni-netd\") pod \"ovnkube-node-8rpk8\" (UID: \"bc5eaee0-b01e-432b-86ae-0d97a04da650\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:02.764128 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.763511 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a6ee59d-3b62-4be1-87e8-79bb3bece069-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l2fgb\" (UID: \"1a6ee59d-3b62-4be1-87e8-79bb3bece069\") " pod="openshift-multus/multus-additional-cni-plugins-l2fgb" Oct 30 12:45:02.764128 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.763549 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2bdf343-b9cd-484a-a8c1-1d0713d74093-lib-modules\") pod \"tuned-js5rp\" (UID: \"b2bdf343-b9cd-484a-a8c1-1d0713d74093\") " pod="openshift-cluster-node-tuning-operator/tuned-js5rp" Oct 30 12:45:02.764128 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.763544 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0-host-slash\") pod \"iptables-alerter-9tp7b\" (UID: \"1758dcf9-60ef-4d0c-9cb2-998bf3b61ff0\") " pod="openshift-network-operator/iptables-alerter-9tp7b" Oct 30 12:45:02.764128 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:02.763656 4931 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 12:45:02.764128 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:02.763781 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-metrics-certs podName:1d06e291-b9d3-48f7-8fc2-f615440631d9 nodeName:}" failed. No retries permitted until 2026-10-30 12:45:03.263745629 +0000 UTC m=+1.867244069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-metrics-certs") pod "router-default-5db587c4bc-dttsw" (UID: "1d06e291-b9d3-48f7-8fc2-f615440631d9") : secret "router-metrics-certs-default" not found Oct 30 12:45:02.825698 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:02.825671 4931 scope.go:122] "RemoveContainer" containerID="c6678b84146d9fb4ce33ee83e569d21905f75a2151e695f43279698e8e6c13a8" Oct 30 12:45:03.267548 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:03.267509 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cb39685-b929-48bd-afd2-228862e62668-metrics-tls\") pod \"dns-default-w52p8\" (UID: \"7cb39685-b929-48bd-afd2-228862e62668\") " pod="openshift-dns/dns-default-w52p8" Oct 30 12:45:03.267726 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:03.267553 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d06e291-b9d3-48f7-8fc2-f615440631d9-service-ca-bundle\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:45:03.267726 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:03.267585 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-metrics-certs\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:45:03.267726 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:03.267608 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-tls\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:45:03.267726 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:03.267632 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-tls\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:45:03.267726 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:03.267647 4931 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 12:45:03.267726 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:03.267663 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-s7c7s\" (UID: \"4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-s7c7s" Oct 30 12:45:03.267726 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:03.267714 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cb39685-b929-48bd-afd2-228862e62668-metrics-tls podName:7cb39685-b929-48bd-afd2-228862e62668 nodeName:}" failed. No retries permitted until 2026-10-30 12:45:04.267694802 +0000 UTC m=+2.871193246 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cb39685-b929-48bd-afd2-228862e62668-metrics-tls") pod "dns-default-w52p8" (UID: "7cb39685-b929-48bd-afd2-228862e62668") : secret "dns-default-metrics-tls" not found Oct 30 12:45:03.268009 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:03.267739 4931 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 12:45:03.268009 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:03.267758 4931 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-596f5cc4c5-sgvjw: secret "image-registry-tls" not found Oct 30 12:45:03.268009 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:03.267757 4931 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 12:45:03.268009 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:03.267760 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/877cfa0b-b1c6-4455-b232-b1ca558b5084-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5bmsj\" (UID: \"877cfa0b-b1c6-4455-b232-b1ca558b5084\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5bmsj" Oct 30 12:45:03.268009 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:03.267767 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1d06e291-b9d3-48f7-8fc2-f615440631d9-service-ca-bundle podName:1d06e291-b9d3-48f7-8fc2-f615440631d9 nodeName:}" failed. No retries permitted until 2026-10-30 12:45:04.2677516 +0000 UTC m=+2.871250023 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1d06e291-b9d3-48f7-8fc2-f615440631d9-service-ca-bundle") pod "router-default-5db587c4bc-dttsw" (UID: "1d06e291-b9d3-48f7-8fc2-f615440631d9") : configmap references non-existent config key: service-ca.crt Oct 30 12:45:03.268009 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:03.267738 4931 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 12:45:03.268009 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:03.267713 4931 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 12:45:03.268009 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:03.267793 4931 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bff985b5d-qwb5l: secret "image-registry-tls" not found Oct 30 12:45:03.268009 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:03.267811 4931 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 12:45:03.268009 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:03.267795 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-tls podName:6000e43a-e175-46fe-b5f7-a417fc1ba5bb nodeName:}" failed. No retries permitted until 2026-10-30 12:45:04.267783598 +0000 UTC m=+2.871282035 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-tls") pod "image-registry-596f5cc4c5-sgvjw" (UID: "6000e43a-e175-46fe-b5f7-a417fc1ba5bb") : secret "image-registry-tls" not found Oct 30 12:45:03.268009 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:03.267859 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-tls podName:4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c nodeName:}" failed. No retries permitted until 2026-10-30 12:45:04.267850582 +0000 UTC m=+2.871349005 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-s7c7s" (UID: "4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c") : secret "samples-operator-tls" not found Oct 30 12:45:03.268009 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:03.267871 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-metrics-certs podName:1d06e291-b9d3-48f7-8fc2-f615440631d9 nodeName:}" failed. No retries permitted until 2026-10-30 12:45:04.267865373 +0000 UTC m=+2.871363796 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-metrics-certs") pod "router-default-5db587c4bc-dttsw" (UID: "1d06e291-b9d3-48f7-8fc2-f615440631d9") : secret "router-metrics-certs-default" not found Oct 30 12:45:03.268009 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:03.267883 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-tls podName:d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5 nodeName:}" failed. No retries permitted until 2026-10-30 12:45:04.267877919 +0000 UTC m=+2.871376342 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-tls") pod "image-registry-7bff985b5d-qwb5l" (UID: "d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5") : secret "image-registry-tls" not found Oct 30 12:45:03.268009 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:03.267893 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/877cfa0b-b1c6-4455-b232-b1ca558b5084-networking-console-plugin-cert podName:877cfa0b-b1c6-4455-b232-b1ca558b5084 nodeName:}" failed. No retries permitted until 2026-10-30 12:45:04.26788897 +0000 UTC m=+2.871387394 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/877cfa0b-b1c6-4455-b232-b1ca558b5084-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5bmsj" (UID: "877cfa0b-b1c6-4455-b232-b1ca558b5084") : secret "networking-console-plugin-cert" not found Oct 30 12:45:03.268009 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:03.267935 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7664727a-bcd2-4169-b70b-a0ecba045321-metrics-certs\") pod \"network-metrics-daemon-j6b2w\" (UID: \"7664727a-bcd2-4169-b70b-a0ecba045321\") " pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:45:03.268009 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:03.267955 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/64bdbc16-e845-45da-8f34-f0d788a75eac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vjdmx\" (UID: \"64bdbc16-e845-45da-8f34-f0d788a75eac\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjdmx" Oct 30 12:45:03.268009 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:03.267982 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40168acf-dcfa-4323-91eb-e710a2b82caa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:45:03.268009 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:03.267999 4931 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 12:45:03.268568 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:03.268025 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7664727a-bcd2-4169-b70b-a0ecba045321-metrics-certs podName:7664727a-bcd2-4169-b70b-a0ecba045321 nodeName:}" failed. No retries permitted until 2026-10-30 12:45:04.268018101 +0000 UTC m=+2.871516526 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7664727a-bcd2-4169-b70b-a0ecba045321-metrics-certs") pod "network-metrics-daemon-j6b2w" (UID: "7664727a-bcd2-4169-b70b-a0ecba045321") : secret "metrics-daemon-secret" not found Oct 30 12:45:03.268568 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:03.268045 4931 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 12:45:03.268568 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:03.268061 4931 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 12:45:03.268568 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:03.268070 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40168acf-dcfa-4323-91eb-e710a2b82caa-insights-runtime-extractor-tls podName:40168acf-dcfa-4323-91eb-e710a2b82caa nodeName:}" failed. No retries permitted until 2026-10-30 12:45:04.268064451 +0000 UTC m=+2.871562874 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40168acf-dcfa-4323-91eb-e710a2b82caa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b94k6" (UID: "40168acf-dcfa-4323-91eb-e710a2b82caa") : secret "insights-runtime-extractor-tls" not found Oct 30 12:45:03.268568 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:03.268081 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64bdbc16-e845-45da-8f34-f0d788a75eac-cluster-monitoring-operator-tls podName:64bdbc16-e845-45da-8f34-f0d788a75eac nodeName:}" failed. No retries permitted until 2026-10-30 12:45:04.268075846 +0000 UTC m=+2.871574270 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/64bdbc16-e845-45da-8f34-f0d788a75eac-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vjdmx" (UID: "64bdbc16-e845-45da-8f34-f0d788a75eac") : secret "cluster-monitoring-operator-tls" not found Oct 30 12:45:03.569014 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:03.568943 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/2.log" Oct 30 12:45:03.569483 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:03.569394 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/1.log" Oct 30 12:45:03.569483 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:03.569433 4931 generic.go:356] "Generic (PLEG): container finished" podID="00ee6aae-7264-4c8d-a337-c7c9f1132e9c" containerID="3582f4305322ba8981537d7c29ce2c78831c9376b60cfa06b5089e960b1356f3" exitCode=255 Oct 30 12:45:03.569581 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:03.569504 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" event={"ID":"00ee6aae-7264-4c8d-a337-c7c9f1132e9c","Type":"ContainerDied","Data":"3582f4305322ba8981537d7c29ce2c78831c9376b60cfa06b5089e960b1356f3"} Oct 30 12:45:03.569581 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:03.569553 4931 scope.go:122] "RemoveContainer" containerID="c6678b84146d9fb4ce33ee83e569d21905f75a2151e695f43279698e8e6c13a8" Oct 30 12:45:03.570028 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:03.570009 4931 scope.go:122] "RemoveContainer" containerID="3582f4305322ba8981537d7c29ce2c78831c9376b60cfa06b5089e960b1356f3" Oct 30 12:45:03.570194 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:03.570175 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-v4mks_openshift-console-operator(00ee6aae-7264-4c8d-a337-c7c9f1132e9c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" podUID="00ee6aae-7264-4c8d-a337-c7c9f1132e9c" Oct 30 12:45:03.775382 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:03.775354 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gt8nx_24a00317-8fb9-41f3-8e1a-d1d985887725/dns-node-resolver/0.log" Oct 30 12:45:04.277635 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:04.277592 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d06e291-b9d3-48f7-8fc2-f615440631d9-service-ca-bundle\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:45:04.277635 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:04.277639 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-metrics-certs\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:45:04.277925 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:04.277661 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-tls\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:45:04.277925 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:04.277679 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-tls\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:45:04.277925 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:04.277708 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-s7c7s\" (UID: \"4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-s7c7s" Oct 30 12:45:04.277925 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:04.277771 4931 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 12:45:04.277925 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:04.277789 4931 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 12:45:04.277925 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:04.277802 4931 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 12:45:04.277925 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:04.277810 4931 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-596f5cc4c5-sgvjw: secret "image-registry-tls" not found Oct 30 12:45:04.277925 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:04.277774 4931 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 12:45:04.277925 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:04.277873 4931 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bff985b5d-qwb5l: secret "image-registry-tls" not found Oct 30 12:45:04.277925 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:04.277873 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/877cfa0b-b1c6-4455-b232-b1ca558b5084-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5bmsj\" (UID: \"877cfa0b-b1c6-4455-b232-b1ca558b5084\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5bmsj" Oct 30 12:45:04.277925 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:04.277896 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-tls podName:6000e43a-e175-46fe-b5f7-a417fc1ba5bb nodeName:}" failed. No retries permitted until 2026-10-30 12:45:06.277877697 +0000 UTC m=+4.881376138 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-tls") pod "image-registry-596f5cc4c5-sgvjw" (UID: "6000e43a-e175-46fe-b5f7-a417fc1ba5bb") : secret "image-registry-tls" not found Oct 30 12:45:04.277925 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:04.277933 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-tls podName:4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c nodeName:}" failed. No retries permitted until 2026-10-30 12:45:06.277916925 +0000 UTC m=+4.881415353 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-s7c7s" (UID: "4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c") : secret "samples-operator-tls" not found Oct 30 12:45:04.278566 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:04.277948 4931 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 12:45:04.278566 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:04.277963 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-metrics-certs podName:1d06e291-b9d3-48f7-8fc2-f615440631d9 nodeName:}" failed. No retries permitted until 2026-10-30 12:45:06.277943522 +0000 UTC m=+4.881441952 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-metrics-certs") pod "router-default-5db587c4bc-dttsw" (UID: "1d06e291-b9d3-48f7-8fc2-f615440631d9") : secret "router-metrics-certs-default" not found Oct 30 12:45:04.278566 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:04.277982 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-tls podName:d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5 nodeName:}" failed. No retries permitted until 2026-10-30 12:45:06.277973163 +0000 UTC m=+4.881471592 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-tls") pod "image-registry-7bff985b5d-qwb5l" (UID: "d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5") : secret "image-registry-tls" not found Oct 30 12:45:04.278566 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:04.278035 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7664727a-bcd2-4169-b70b-a0ecba045321-metrics-certs\") pod \"network-metrics-daemon-j6b2w\" (UID: \"7664727a-bcd2-4169-b70b-a0ecba045321\") " pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:45:04.278566 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:04.278055 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/877cfa0b-b1c6-4455-b232-b1ca558b5084-networking-console-plugin-cert podName:877cfa0b-b1c6-4455-b232-b1ca558b5084 nodeName:}" failed. No retries permitted until 2026-10-30 12:45:06.278044215 +0000 UTC m=+4.881542651 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/877cfa0b-b1c6-4455-b232-b1ca558b5084-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5bmsj" (UID: "877cfa0b-b1c6-4455-b232-b1ca558b5084") : secret "networking-console-plugin-cert" not found Oct 30 12:45:04.278566 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:04.278096 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/64bdbc16-e845-45da-8f34-f0d788a75eac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vjdmx\" (UID: \"64bdbc16-e845-45da-8f34-f0d788a75eac\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjdmx" Oct 30 12:45:04.278566 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:04.278129 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40168acf-dcfa-4323-91eb-e710a2b82caa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:45:04.278566 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:04.278191 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cb39685-b929-48bd-afd2-228862e62668-metrics-tls\") pod \"dns-default-w52p8\" (UID: \"7cb39685-b929-48bd-afd2-228862e62668\") " pod="openshift-dns/dns-default-w52p8" Oct 30 12:45:04.278566 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:04.278306 4931 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 12:45:04.278566 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:04.278362 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40168acf-dcfa-4323-91eb-e710a2b82caa-insights-runtime-extractor-tls podName:40168acf-dcfa-4323-91eb-e710a2b82caa nodeName:}" failed. No retries permitted until 2026-10-30 12:45:06.278348959 +0000 UTC m=+4.881847398 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40168acf-dcfa-4323-91eb-e710a2b82caa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b94k6" (UID: "40168acf-dcfa-4323-91eb-e710a2b82caa") : secret "insights-runtime-extractor-tls" not found Oct 30 12:45:04.278566 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:04.278312 4931 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 12:45:04.278566 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:04.278406 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cb39685-b929-48bd-afd2-228862e62668-metrics-tls podName:7cb39685-b929-48bd-afd2-228862e62668 nodeName:}" failed. No retries permitted until 2026-10-30 12:45:06.278395777 +0000 UTC m=+4.881894206 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cb39685-b929-48bd-afd2-228862e62668-metrics-tls") pod "dns-default-w52p8" (UID: "7cb39685-b929-48bd-afd2-228862e62668") : secret "dns-default-metrics-tls" not found Oct 30 12:45:04.278566 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:04.278411 4931 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 12:45:04.278566 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:04.278488 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d06e291-b9d3-48f7-8fc2-f615440631d9-service-ca-bundle\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:45:04.280526 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:04.280502 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7664727a-bcd2-4169-b70b-a0ecba045321-metrics-certs\") pod \"network-metrics-daemon-j6b2w\" (UID: \"7664727a-bcd2-4169-b70b-a0ecba045321\") " pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:45:04.280596 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:04.280574 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/64bdbc16-e845-45da-8f34-f0d788a75eac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vjdmx\" (UID: \"64bdbc16-e845-45da-8f34-f0d788a75eac\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjdmx" Oct 30 12:45:04.324634 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:04.324607 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjdmx" Oct 30 12:45:04.333009 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:04.332983 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6b2w" Oct 30 12:45:04.574469 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:04.574392 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/2.log" Oct 30 12:45:04.627253 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:04.627224 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-vjdmx"] Oct 30 12:45:04.630309 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:04.630258 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64bdbc16_e845_45da_8f34_f0d788a75eac.slice/crio-c719b6fc5aefe578415092aef65b2be40cf4bbbd936806bb137f30b7f01b6d32 WatchSource:0}: Error finding container c719b6fc5aefe578415092aef65b2be40cf4bbbd936806bb137f30b7f01b6d32: Status 404 returned error can't find the container with id c719b6fc5aefe578415092aef65b2be40cf4bbbd936806bb137f30b7f01b6d32 Oct 30 12:45:04.632136 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:04.632116 4931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:45:04.655360 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:04.655336 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-j6b2w"] Oct 30 12:45:04.658516 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:04.658493 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7664727a_bcd2_4169_b70b_a0ecba045321.slice/crio-b7978a6b6e738d8e292382fa6b7213b9ecbec8e7748104d6bb06c9ab7d7cf4ea WatchSource:0}: Error finding container b7978a6b6e738d8e292382fa6b7213b9ecbec8e7748104d6bb06c9ab7d7cf4ea: Status 404 returned error can't find the container with id b7978a6b6e738d8e292382fa6b7213b9ecbec8e7748104d6bb06c9ab7d7cf4ea Oct 30 12:45:04.972855 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:04.972827 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rjhg6_49c39196-4e77-4bcf-86b1-705a263e394b/node-ca/0.log" Oct 30 12:45:05.581088 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:05.581044 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j6b2w" event={"ID":"7664727a-bcd2-4169-b70b-a0ecba045321","Type":"ContainerStarted","Data":"b7978a6b6e738d8e292382fa6b7213b9ecbec8e7748104d6bb06c9ab7d7cf4ea"} Oct 30 12:45:05.583302 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:05.583257 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjdmx" event={"ID":"64bdbc16-e845-45da-8f34-f0d788a75eac","Type":"ContainerStarted","Data":"c719b6fc5aefe578415092aef65b2be40cf4bbbd936806bb137f30b7f01b6d32"} Oct 30 12:45:05.671920 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:05.671894 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" Oct 30 12:45:05.672347 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:05.672314 4931 scope.go:122] "RemoveContainer" containerID="3582f4305322ba8981537d7c29ce2c78831c9376b60cfa06b5089e960b1356f3" Oct 30 12:45:05.672559 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:05.672533 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-v4mks_openshift-console-operator(00ee6aae-7264-4c8d-a337-c7c9f1132e9c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" podUID="00ee6aae-7264-4c8d-a337-c7c9f1132e9c" Oct 30 12:45:05.774080 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:05.773998 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dnpgs_4df70e16-5acb-494e-ab67-855c809e3411/serve-healthcheck-canary/0.log" Oct 30 12:45:05.973704 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:05.973670 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-4h8hg_b32002db-049b-4456-91d7-66c33a85c040/migrator/0.log" Oct 30 12:45:06.043503 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:06.043466 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ljvqz" Oct 30 12:45:06.299097 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:06.299062 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/877cfa0b-b1c6-4455-b232-b1ca558b5084-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5bmsj\" (UID: \"877cfa0b-b1c6-4455-b232-b1ca558b5084\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5bmsj" Oct 30 12:45:06.299297 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:06.299120 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40168acf-dcfa-4323-91eb-e710a2b82caa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:45:06.299297 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:06.299146 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cb39685-b929-48bd-afd2-228862e62668-metrics-tls\") pod \"dns-default-w52p8\" (UID: \"7cb39685-b929-48bd-afd2-228862e62668\") " pod="openshift-dns/dns-default-w52p8" Oct 30 12:45:06.299297 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:06.299169 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-metrics-certs\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:45:06.299297 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:06.299188 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-tls\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:45:06.299297 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:06.299238 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-tls\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:45:06.299561 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:06.299303 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-s7c7s\" (UID: \"4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-s7c7s" Oct 30 12:45:06.299561 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:06.299248 4931 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 12:45:06.299561 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:06.299388 4931 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 12:45:06.299561 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:06.299415 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40168acf-dcfa-4323-91eb-e710a2b82caa-insights-runtime-extractor-tls podName:40168acf-dcfa-4323-91eb-e710a2b82caa nodeName:}" failed. No retries permitted until 2026-10-30 12:45:10.299395737 +0000 UTC m=+8.902894185 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40168acf-dcfa-4323-91eb-e710a2b82caa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b94k6" (UID: "40168acf-dcfa-4323-91eb-e710a2b82caa") : secret "insights-runtime-extractor-tls" not found Oct 30 12:45:06.299561 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:06.299426 4931 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 12:45:06.299561 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:06.299456 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cb39685-b929-48bd-afd2-228862e62668-metrics-tls podName:7cb39685-b929-48bd-afd2-228862e62668 nodeName:}" failed. No retries permitted until 2026-10-30 12:45:10.29943458 +0000 UTC m=+8.902933023 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cb39685-b929-48bd-afd2-228862e62668-metrics-tls") pod "dns-default-w52p8" (UID: "7cb39685-b929-48bd-afd2-228862e62668") : secret "dns-default-metrics-tls" not found Oct 30 12:45:06.299561 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:06.299319 4931 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 12:45:06.299561 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:06.299482 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-tls podName:4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c nodeName:}" failed. No retries permitted until 2026-10-30 12:45:10.299468957 +0000 UTC m=+8.902967392 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-s7c7s" (UID: "4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c") : secret "samples-operator-tls" not found Oct 30 12:45:06.299561 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:06.299503 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-metrics-certs podName:1d06e291-b9d3-48f7-8fc2-f615440631d9 nodeName:}" failed. No retries permitted until 2026-10-30 12:45:10.299492391 +0000 UTC m=+8.902990842 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-metrics-certs") pod "router-default-5db587c4bc-dttsw" (UID: "1d06e291-b9d3-48f7-8fc2-f615440631d9") : secret "router-metrics-certs-default" not found Oct 30 12:45:06.301916 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:06.301895 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-tls\") pod \"image-registry-7bff985b5d-qwb5l\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:45:06.302018 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:06.301911 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-tls\") pod \"image-registry-596f5cc4c5-sgvjw\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:45:06.302063 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:06.302019 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/877cfa0b-b1c6-4455-b232-b1ca558b5084-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5bmsj\" (UID: \"877cfa0b-b1c6-4455-b232-b1ca558b5084\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5bmsj" Oct 30 12:45:06.332446 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:06.332419 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:06.374850 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:06.374789 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-pptmg_71e82393-029b-4388-86df-8bd068720867/kube-storage-version-migrator-operator/0.log" Oct 30 12:45:06.425314 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:06.425287 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:45:06.431362 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:06.431341 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-5bmsj" Oct 30 12:45:06.436620 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:06.436591 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:45:06.588158 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:06.588083 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j6b2w" event={"ID":"7664727a-bcd2-4169-b70b-a0ecba045321","Type":"ContainerStarted","Data":"b008479cf3345f866a7734500a303f94083e539788c816ee955c70220821759b"} Oct 30 12:45:06.588158 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:06.588145 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j6b2w" event={"ID":"7664727a-bcd2-4169-b70b-a0ecba045321","Type":"ContainerStarted","Data":"0a130e2a5e39bb64e647650fd72cf37b46684903b0b4439e26e1bdf21aba6246"} Oct 30 12:45:06.589732 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:06.589700 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjdmx" event={"ID":"64bdbc16-e845-45da-8f34-f0d788a75eac","Type":"ContainerStarted","Data":"eef03cbde8d9789d4e338474fc939fff8a59d1c0a6f08b8432785b2f83cd1dce"} Oct 30 12:45:06.746658 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:06.746623 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-5bmsj"] Oct 30 12:45:06.749931 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:06.749904 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod877cfa0b_b1c6_4455_b232_b1ca558b5084.slice/crio-fd488b049c92fa916585f1abc89b7b891d677f84f0a8b444a3f87bf53cb7b3c4 WatchSource:0}: Error finding container fd488b049c92fa916585f1abc89b7b891d677f84f0a8b444a3f87bf53cb7b3c4: Status 404 returned error can't find the container with id fd488b049c92fa916585f1abc89b7b891d677f84f0a8b444a3f87bf53cb7b3c4 Oct 30 12:45:06.758966 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:06.758946 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bff985b5d-qwb5l"] Oct 30 12:45:06.761293 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:06.761246 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8d7b93d_1b9a_4c2e_b80b_ddeb336f48a5.slice/crio-788c8feb3a6101ec87781cf91e3f293764b1b54bb683b9d57e1f5aadfb00668f WatchSource:0}: Error finding container 788c8feb3a6101ec87781cf91e3f293764b1b54bb683b9d57e1f5aadfb00668f: Status 404 returned error can't find the container with id 788c8feb3a6101ec87781cf91e3f293764b1b54bb683b9d57e1f5aadfb00668f Oct 30 12:45:06.763406 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:06.763381 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-596f5cc4c5-sgvjw"] Oct 30 12:45:06.766122 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:06.766100 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6000e43a_e175_46fe_b5f7_a417fc1ba5bb.slice/crio-93c9e8970612a0952d3376de288d80ab27f9eb95263ff91dc918c2fd8e598ef3 WatchSource:0}: Error finding container 93c9e8970612a0952d3376de288d80ab27f9eb95263ff91dc918c2fd8e598ef3: Status 404 returned error can't find the container with id 93c9e8970612a0952d3376de288d80ab27f9eb95263ff91dc918c2fd8e598ef3 Oct 30 12:45:07.333164 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:07.333130 4931 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" Oct 30 12:45:07.333598 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:07.333579 4931 scope.go:122] "RemoveContainer" containerID="3582f4305322ba8981537d7c29ce2c78831c9376b60cfa06b5089e960b1356f3" Oct 30 12:45:07.333848 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:07.333796 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-v4mks_openshift-console-operator(00ee6aae-7264-4c8d-a337-c7c9f1132e9c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" podUID="00ee6aae-7264-4c8d-a337-c7c9f1132e9c" Oct 30 12:45:07.595829 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:07.595395 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" event={"ID":"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5","Type":"ContainerStarted","Data":"ab6fff72e975fdabab14ac7131ecc78b6e8a72160c2d2768ba55327f238d71e9"} Oct 30 12:45:07.595829 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:07.595457 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" event={"ID":"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5","Type":"ContainerStarted","Data":"788c8feb3a6101ec87781cf91e3f293764b1b54bb683b9d57e1f5aadfb00668f"} Oct 30 12:45:07.597282 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:07.597233 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" event={"ID":"6000e43a-e175-46fe-b5f7-a417fc1ba5bb","Type":"ContainerStarted","Data":"468c8e9f73ad8f848c42905e9a4e32c481a9cbfde9a71cf4a329b0f543d311fd"} Oct 30 12:45:07.597426 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:07.597290 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" event={"ID":"6000e43a-e175-46fe-b5f7-a417fc1ba5bb","Type":"ContainerStarted","Data":"93c9e8970612a0952d3376de288d80ab27f9eb95263ff91dc918c2fd8e598ef3"} Oct 30 12:45:07.598545 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:07.598520 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-5bmsj" event={"ID":"877cfa0b-b1c6-4455-b232-b1ca558b5084","Type":"ContainerStarted","Data":"fd488b049c92fa916585f1abc89b7b891d677f84f0a8b444a3f87bf53cb7b3c4"} Oct 30 12:45:07.723521 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:07.723491 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-k2fjm" Oct 30 12:45:08.603748 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:08.603712 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-5bmsj" event={"ID":"877cfa0b-b1c6-4455-b232-b1ca558b5084","Type":"ContainerStarted","Data":"256fca6d002ca89367ca469257982209767c51a428ea4a949623b9d3c9631bf0"} Oct 30 12:45:08.603748 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:08.603736 4931 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 12:45:08.603748 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:08.603756 4931 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 12:45:09.884209 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:09.884179 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:09.884607 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:09.884496 4931 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 12:45:10.333323 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:10.333285 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cb39685-b929-48bd-afd2-228862e62668-metrics-tls\") pod \"dns-default-w52p8\" (UID: \"7cb39685-b929-48bd-afd2-228862e62668\") " pod="openshift-dns/dns-default-w52p8" Oct 30 12:45:10.333499 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:10.333331 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-metrics-certs\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:45:10.333499 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:10.333369 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-s7c7s\" (UID: \"4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-s7c7s" Oct 30 12:45:10.333499 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:10.333433 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40168acf-dcfa-4323-91eb-e710a2b82caa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:45:10.333499 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:10.333493 4931 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 12:45:10.333636 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:10.333515 4931 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 12:45:10.333636 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:10.333555 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-metrics-certs podName:1d06e291-b9d3-48f7-8fc2-f615440631d9 nodeName:}" failed. No retries permitted until 2026-10-30 12:45:18.333539726 +0000 UTC m=+16.937038167 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-metrics-certs") pod "router-default-5db587c4bc-dttsw" (UID: "1d06e291-b9d3-48f7-8fc2-f615440631d9") : secret "router-metrics-certs-default" not found Oct 30 12:45:10.333636 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:10.333580 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40168acf-dcfa-4323-91eb-e710a2b82caa-insights-runtime-extractor-tls podName:40168acf-dcfa-4323-91eb-e710a2b82caa nodeName:}" failed. No retries permitted until 2026-10-30 12:45:18.333561278 +0000 UTC m=+16.937059702 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40168acf-dcfa-4323-91eb-e710a2b82caa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b94k6" (UID: "40168acf-dcfa-4323-91eb-e710a2b82caa") : secret "insights-runtime-extractor-tls" not found Oct 30 12:45:10.335745 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:10.335713 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cb39685-b929-48bd-afd2-228862e62668-metrics-tls\") pod \"dns-default-w52p8\" (UID: \"7cb39685-b929-48bd-afd2-228862e62668\") " pod="openshift-dns/dns-default-w52p8" Oct 30 12:45:10.335863 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:10.335746 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-s7c7s\" (UID: \"4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-s7c7s" Oct 30 12:45:10.337642 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:10.337626 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w52p8" Oct 30 12:45:10.625434 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:10.625358 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-s7c7s" Oct 30 12:45:10.637181 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:10.636851 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w52p8"] Oct 30 12:45:10.640990 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:10.640950 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb39685_b929_48bd_afd2_228862e62668.slice/crio-cf2c065316988dee11f26609977ce458f00a2bf28c19cad48f5517e71d14e8d3 WatchSource:0}: Error finding container cf2c065316988dee11f26609977ce458f00a2bf28c19cad48f5517e71d14e8d3: Status 404 returned error can't find the container with id cf2c065316988dee11f26609977ce458f00a2bf28c19cad48f5517e71d14e8d3 Oct 30 12:45:10.924519 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:10.924439 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-s7c7s"] Oct 30 12:45:11.553137 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:11.553100 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:45:11.615802 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:11.615762 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-s7c7s" event={"ID":"4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c","Type":"ContainerStarted","Data":"2be5256fa53343211f31c7af40d64cb1fc4330e9001327adcc32977cb94fc61e"} Oct 30 12:45:11.617521 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:11.617483 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w52p8" event={"ID":"7cb39685-b929-48bd-afd2-228862e62668","Type":"ContainerStarted","Data":"cf2c065316988dee11f26609977ce458f00a2bf28c19cad48f5517e71d14e8d3"} Oct 30 12:45:12.623902 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:12.623527 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w52p8" event={"ID":"7cb39685-b929-48bd-afd2-228862e62668","Type":"ContainerStarted","Data":"2b76b88c268d828fe3c255487af1660b39a3a2e6e11b35a5ea88d4c7ec276b98"} Oct 30 12:45:12.624394 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:12.623919 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w52p8" event={"ID":"7cb39685-b929-48bd-afd2-228862e62668","Type":"ContainerStarted","Data":"3dffa63190ad9c9887ca402d2debf8201978cba9c34ce0ed8ccb5f38756ca5c2"} Oct 30 12:45:12.624394 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:12.624348 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-w52p8" Oct 30 12:45:13.506055 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:13.506019 4931 scope.go:122] "RemoveContainer" containerID="3582f4305322ba8981537d7c29ce2c78831c9376b60cfa06b5089e960b1356f3" Oct 30 12:45:13.629531 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:13.629502 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/2.log" Oct 30 12:45:13.629943 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:13.629621 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" event={"ID":"00ee6aae-7264-4c8d-a337-c7c9f1132e9c","Type":"ContainerStarted","Data":"670fd38dd4b266c8b423c08bd62df2a123b592b203e71862bd41243801f4663f"} Oct 30 12:45:13.629943 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:13.629913 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" Oct 30 12:45:13.631033 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:13.631007 4931 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-v4mks container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.15:8443/readyz\": dial tcp 10.132.0.15:8443: connect: connection refused" start-of-body= Oct 30 12:45:13.631176 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:13.631053 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" podUID="00ee6aae-7264-4c8d-a337-c7c9f1132e9c" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.15:8443/readyz\": dial tcp 10.132.0.15:8443: connect: connection refused" Oct 30 12:45:13.631671 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:13.631648 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-s7c7s" event={"ID":"4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c","Type":"ContainerStarted","Data":"6995e407de03afe38fb87335f6e08a882ca48da6b74c78dd8994c5b09a48d3a7"} Oct 30 12:45:13.631785 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:13.631693 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-s7c7s" event={"ID":"4b715e45-2ee3-4d46-8ecc-50eea9f3dd2c","Type":"ContainerStarted","Data":"4adc80176826b3ab4e6ac63473aa3a8559d9bcfd96f4b038980e41997544e460"} Oct 30 12:45:14.639502 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:14.639474 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-v4mks" Oct 30 12:45:15.915060 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:15.915024 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:45:18.407713 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:18.407682 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40168acf-dcfa-4323-91eb-e710a2b82caa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:45:18.408206 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:18.407723 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-metrics-certs\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:45:18.410023 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:18.409998 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d06e291-b9d3-48f7-8fc2-f615440631d9-metrics-certs\") pod \"router-default-5db587c4bc-dttsw\" (UID: \"1d06e291-b9d3-48f7-8fc2-f615440631d9\") " pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:45:18.410023 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:18.410011 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40168acf-dcfa-4323-91eb-e710a2b82caa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b94k6\" (UID: \"40168acf-dcfa-4323-91eb-e710a2b82caa\") " pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:45:18.431018 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:18.430990 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:45:18.439082 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:18.439063 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b94k6" Oct 30 12:45:18.761084 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:18.761058 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b94k6"] Oct 30 12:45:18.763798 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:18.763767 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40168acf_dcfa_4323_91eb_e710a2b82caa.slice/crio-11fa6b3f1b58095998af7367fc3688c93107e8847e6351682d99a4266134cb1d WatchSource:0}: Error finding container 11fa6b3f1b58095998af7367fc3688c93107e8847e6351682d99a4266134cb1d: Status 404 returned error can't find the container with id 11fa6b3f1b58095998af7367fc3688c93107e8847e6351682d99a4266134cb1d Oct 30 12:45:18.770139 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:18.770114 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d06e291_b9d3_48f7_8fc2_f615440631d9.slice/crio-263ac5a3401576d27350b49cd100eb7200bb96f66d415c62343d84ea0a2ddff7 WatchSource:0}: Error finding container 263ac5a3401576d27350b49cd100eb7200bb96f66d415c62343d84ea0a2ddff7: Status 404 returned error can't find the container with id 263ac5a3401576d27350b49cd100eb7200bb96f66d415c62343d84ea0a2ddff7 Oct 30 12:45:18.774995 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:18.774976 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5db587c4bc-dttsw"] Oct 30 12:45:19.656295 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:19.656246 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5db587c4bc-dttsw" event={"ID":"1d06e291-b9d3-48f7-8fc2-f615440631d9","Type":"ContainerStarted","Data":"f15dca7efa64efca7581689f19d3710d90fd59382a4de8a4b36a9c08bed4e74e"} Oct 30 12:45:19.656654 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:19.656320 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5db587c4bc-dttsw" event={"ID":"1d06e291-b9d3-48f7-8fc2-f615440631d9","Type":"ContainerStarted","Data":"263ac5a3401576d27350b49cd100eb7200bb96f66d415c62343d84ea0a2ddff7"} Oct 30 12:45:19.662171 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:19.662147 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b94k6" event={"ID":"40168acf-dcfa-4323-91eb-e710a2b82caa","Type":"ContainerStarted","Data":"d3e70eaf704798ec31e7d321f3563f4c475f9fb452c14d30383430c85fed7420"} Oct 30 12:45:19.662329 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:19.662229 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b94k6" event={"ID":"40168acf-dcfa-4323-91eb-e710a2b82caa","Type":"ContainerStarted","Data":"94744501e3ebdd901db0455a582a48e8d44183c0625deef15437581391bed824"} Oct 30 12:45:19.662329 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:19.662261 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b94k6" event={"ID":"40168acf-dcfa-4323-91eb-e710a2b82caa","Type":"ContainerStarted","Data":"11fa6b3f1b58095998af7367fc3688c93107e8847e6351682d99a4266134cb1d"} Oct 30 12:45:20.245572 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:20.245541 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8rpk8" Oct 30 12:45:20.431506 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:20.431471 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:45:20.434004 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:20.433978 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:45:20.479257 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:20.478985 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7bff985b5d-qwb5l"] Oct 30 12:45:20.667013 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:20.666939 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:45:20.668233 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:20.668213 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5db587c4bc-dttsw" Oct 30 12:45:21.557417 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:21.557385 4931 patch_prober.go:29] interesting pod/image-registry-596f5cc4c5-sgvjw container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 12:45:21.557576 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:21.557438 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" podUID="6000e43a-e175-46fe-b5f7-a417fc1ba5bb" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:45:21.672083 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:21.672049 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b94k6" event={"ID":"40168acf-dcfa-4323-91eb-e710a2b82caa","Type":"ContainerStarted","Data":"f664637aa3e538c862e4ea946a98cf96f00060c9b15847732feccd126d968a07"} Oct 30 12:45:22.633659 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:22.633575 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-w52p8" Oct 30 12:45:25.360763 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:25.360724 4931 patch_prober.go:29] interesting pod/image-registry-596f5cc4c5-sgvjw container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 12:45:25.361225 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:25.360780 4931 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" podUID="6000e43a-e175-46fe-b5f7-a417fc1ba5bb" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:45:29.245446 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.245414 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qlkj4"] Oct 30 12:45:29.279251 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.279226 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qlkj4" Oct 30 12:45:29.312472 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.312439 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 12:45:29.312625 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.312613 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 12:45:29.314970 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.314954 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-njv68\"" Oct 30 12:45:29.328051 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.328025 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 12:45:29.328163 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.328080 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 12:45:29.397053 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.397018 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91a082d9-7a7e-401e-9c9a-3615a462c3a1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qlkj4\" (UID: \"91a082d9-7a7e-401e-9c9a-3615a462c3a1\") " pod="openshift-monitoring/node-exporter-qlkj4" Oct 30 12:45:29.397053 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.397055 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/91a082d9-7a7e-401e-9c9a-3615a462c3a1-node-exporter-accelerators-collector-config\") pod \"node-exporter-qlkj4\" (UID: \"91a082d9-7a7e-401e-9c9a-3615a462c3a1\") " pod="openshift-monitoring/node-exporter-qlkj4" Oct 30 12:45:29.397288 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.397099 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fs8n\" (UniqueName: \"kubernetes.io/projected/91a082d9-7a7e-401e-9c9a-3615a462c3a1-kube-api-access-2fs8n\") pod \"node-exporter-qlkj4\" (UID: \"91a082d9-7a7e-401e-9c9a-3615a462c3a1\") " pod="openshift-monitoring/node-exporter-qlkj4" Oct 30 12:45:29.397288 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.397157 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/91a082d9-7a7e-401e-9c9a-3615a462c3a1-node-exporter-wtmp\") pod \"node-exporter-qlkj4\" (UID: \"91a082d9-7a7e-401e-9c9a-3615a462c3a1\") " pod="openshift-monitoring/node-exporter-qlkj4" Oct 30 12:45:29.397288 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.397183 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91a082d9-7a7e-401e-9c9a-3615a462c3a1-sys\") pod \"node-exporter-qlkj4\" (UID: \"91a082d9-7a7e-401e-9c9a-3615a462c3a1\") " pod="openshift-monitoring/node-exporter-qlkj4" Oct 30 12:45:29.397288 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.397201 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/91a082d9-7a7e-401e-9c9a-3615a462c3a1-node-exporter-tls\") pod \"node-exporter-qlkj4\" (UID: \"91a082d9-7a7e-401e-9c9a-3615a462c3a1\") " pod="openshift-monitoring/node-exporter-qlkj4" Oct 30 12:45:29.397416 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.397298 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/91a082d9-7a7e-401e-9c9a-3615a462c3a1-node-exporter-textfile\") pod \"node-exporter-qlkj4\" (UID: \"91a082d9-7a7e-401e-9c9a-3615a462c3a1\") " pod="openshift-monitoring/node-exporter-qlkj4" Oct 30 12:45:29.397416 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.397371 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/91a082d9-7a7e-401e-9c9a-3615a462c3a1-root\") pod \"node-exporter-qlkj4\" (UID: \"91a082d9-7a7e-401e-9c9a-3615a462c3a1\") " pod="openshift-monitoring/node-exporter-qlkj4" Oct 30 12:45:29.397416 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.397389 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91a082d9-7a7e-401e-9c9a-3615a462c3a1-metrics-client-ca\") pod \"node-exporter-qlkj4\" (UID: \"91a082d9-7a7e-401e-9c9a-3615a462c3a1\") " pod="openshift-monitoring/node-exporter-qlkj4" Oct 30 12:45:29.498311 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.498205 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/91a082d9-7a7e-401e-9c9a-3615a462c3a1-node-exporter-textfile\") pod \"node-exporter-qlkj4\" (UID: \"91a082d9-7a7e-401e-9c9a-3615a462c3a1\") " pod="openshift-monitoring/node-exporter-qlkj4" Oct 30 12:45:29.498311 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.498291 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/91a082d9-7a7e-401e-9c9a-3615a462c3a1-root\") pod \"node-exporter-qlkj4\" (UID: \"91a082d9-7a7e-401e-9c9a-3615a462c3a1\") " pod="openshift-monitoring/node-exporter-qlkj4" Oct 30 12:45:29.498539 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.498318 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91a082d9-7a7e-401e-9c9a-3615a462c3a1-metrics-client-ca\") pod \"node-exporter-qlkj4\" (UID: \"91a082d9-7a7e-401e-9c9a-3615a462c3a1\") " pod="openshift-monitoring/node-exporter-qlkj4" Oct 30 12:45:29.498539 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.498352 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91a082d9-7a7e-401e-9c9a-3615a462c3a1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qlkj4\" (UID: \"91a082d9-7a7e-401e-9c9a-3615a462c3a1\") " pod="openshift-monitoring/node-exporter-qlkj4" Oct 30 12:45:29.498539 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.498401 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/91a082d9-7a7e-401e-9c9a-3615a462c3a1-node-exporter-accelerators-collector-config\") pod \"node-exporter-qlkj4\" (UID: \"91a082d9-7a7e-401e-9c9a-3615a462c3a1\") " pod="openshift-monitoring/node-exporter-qlkj4" Oct 30 12:45:29.498539 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.498422 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/91a082d9-7a7e-401e-9c9a-3615a462c3a1-root\") pod \"node-exporter-qlkj4\" (UID: \"91a082d9-7a7e-401e-9c9a-3615a462c3a1\") " pod="openshift-monitoring/node-exporter-qlkj4" Oct 30 12:45:29.498539 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.498452 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fs8n\" (UniqueName: \"kubernetes.io/projected/91a082d9-7a7e-401e-9c9a-3615a462c3a1-kube-api-access-2fs8n\") pod \"node-exporter-qlkj4\" (UID: \"91a082d9-7a7e-401e-9c9a-3615a462c3a1\") " pod="openshift-monitoring/node-exporter-qlkj4" Oct 30 12:45:29.498539 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.498487 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/91a082d9-7a7e-401e-9c9a-3615a462c3a1-node-exporter-wtmp\") pod \"node-exporter-qlkj4\" (UID: \"91a082d9-7a7e-401e-9c9a-3615a462c3a1\") " pod="openshift-monitoring/node-exporter-qlkj4" Oct 30 12:45:29.498539 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.498518 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91a082d9-7a7e-401e-9c9a-3615a462c3a1-sys\") pod \"node-exporter-qlkj4\" (UID: \"91a082d9-7a7e-401e-9c9a-3615a462c3a1\") " pod="openshift-monitoring/node-exporter-qlkj4" Oct 30 12:45:29.498539 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.498540 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/91a082d9-7a7e-401e-9c9a-3615a462c3a1-node-exporter-tls\") pod \"node-exporter-qlkj4\" (UID: \"91a082d9-7a7e-401e-9c9a-3615a462c3a1\") " pod="openshift-monitoring/node-exporter-qlkj4" Oct 30 12:45:29.498927 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.498685 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/91a082d9-7a7e-401e-9c9a-3615a462c3a1-node-exporter-textfile\") pod \"node-exporter-qlkj4\" (UID: \"91a082d9-7a7e-401e-9c9a-3615a462c3a1\") " pod="openshift-monitoring/node-exporter-qlkj4" Oct 30 12:45:29.498927 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.498843 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/91a082d9-7a7e-401e-9c9a-3615a462c3a1-node-exporter-wtmp\") pod \"node-exporter-qlkj4\" (UID: \"91a082d9-7a7e-401e-9c9a-3615a462c3a1\") " pod="openshift-monitoring/node-exporter-qlkj4" Oct 30 12:45:29.498927 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.498886 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91a082d9-7a7e-401e-9c9a-3615a462c3a1-sys\") pod \"node-exporter-qlkj4\" (UID: \"91a082d9-7a7e-401e-9c9a-3615a462c3a1\") " pod="openshift-monitoring/node-exporter-qlkj4" Oct 30 12:45:29.499090 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.498965 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91a082d9-7a7e-401e-9c9a-3615a462c3a1-metrics-client-ca\") pod \"node-exporter-qlkj4\" (UID: \"91a082d9-7a7e-401e-9c9a-3615a462c3a1\") " pod="openshift-monitoring/node-exporter-qlkj4" Oct 30 12:45:29.499090 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.499034 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/91a082d9-7a7e-401e-9c9a-3615a462c3a1-node-exporter-accelerators-collector-config\") pod \"node-exporter-qlkj4\" (UID: \"91a082d9-7a7e-401e-9c9a-3615a462c3a1\") " pod="openshift-monitoring/node-exporter-qlkj4" Oct 30 12:45:29.500903 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.500875 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91a082d9-7a7e-401e-9c9a-3615a462c3a1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qlkj4\" (UID: \"91a082d9-7a7e-401e-9c9a-3615a462c3a1\") " pod="openshift-monitoring/node-exporter-qlkj4" Oct 30 12:45:29.501003 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.500957 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/91a082d9-7a7e-401e-9c9a-3615a462c3a1-node-exporter-tls\") pod \"node-exporter-qlkj4\" (UID: \"91a082d9-7a7e-401e-9c9a-3615a462c3a1\") " pod="openshift-monitoring/node-exporter-qlkj4" Oct 30 12:45:29.523831 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.523804 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fs8n\" (UniqueName: \"kubernetes.io/projected/91a082d9-7a7e-401e-9c9a-3615a462c3a1-kube-api-access-2fs8n\") pod \"node-exporter-qlkj4\" (UID: \"91a082d9-7a7e-401e-9c9a-3615a462c3a1\") " pod="openshift-monitoring/node-exporter-qlkj4" Oct 30 12:45:29.629810 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.629775 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qlkj4" Oct 30 12:45:29.672023 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:45:29.671997 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91a082d9_7a7e_401e_9c9a_3615a462c3a1.slice/crio-9894add60198ca83b81c0027a440b25e7fdb1a321c5428337898c29a2aa2118a WatchSource:0}: Error finding container 9894add60198ca83b81c0027a440b25e7fdb1a321c5428337898c29a2aa2118a: Status 404 returned error can't find the container with id 9894add60198ca83b81c0027a440b25e7fdb1a321c5428337898c29a2aa2118a Oct 30 12:45:29.705102 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:29.705070 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qlkj4" event={"ID":"91a082d9-7a7e-401e-9c9a-3615a462c3a1","Type":"ContainerStarted","Data":"9894add60198ca83b81c0027a440b25e7fdb1a321c5428337898c29a2aa2118a"} Oct 30 12:45:30.485872 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:30.485841 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:45:31.558135 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:31.558107 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:45:31.714938 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:31.714898 4931 generic.go:356] "Generic (PLEG): container finished" podID="91a082d9-7a7e-401e-9c9a-3615a462c3a1" containerID="2b04abb1ed88d6b8efaf94da20c5b83a8ca54c37bf5b145031da7542345d6a4a" exitCode=0 Oct 30 12:45:31.715127 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:31.714944 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qlkj4" event={"ID":"91a082d9-7a7e-401e-9c9a-3615a462c3a1","Type":"ContainerDied","Data":"2b04abb1ed88d6b8efaf94da20c5b83a8ca54c37bf5b145031da7542345d6a4a"} Oct 30 12:45:32.724340 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:32.724302 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qlkj4" event={"ID":"91a082d9-7a7e-401e-9c9a-3615a462c3a1","Type":"ContainerStarted","Data":"8f4e9c7f24fb648753f9744e40be6beea74ed32fb476b0ce4c51937c28196fa4"} Oct 30 12:45:32.724726 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:32.724386 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qlkj4" event={"ID":"91a082d9-7a7e-401e-9c9a-3615a462c3a1","Type":"ContainerStarted","Data":"68d63be8305af88e053bb152248f58fbb30451208111c5ced9aab00810194aea"} Oct 30 12:45:32.772922 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:32.772853 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qlkj4" podStartSLOduration=2.263338739 podStartE2EDuration="3.772835119s" totalImagesPullingTime="1.428712149s" totalInitContainerRuntime="80.784231ms" isStatefulPod=true podCreationTimestamp="2026-10-30 12:45:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:32.771773265 +0000 UTC m=+31.375271720" watchObservedRunningTime="2026-10-30 12:45:32.772835119 +0000 UTC m=+31.376333566" Oct 30 12:45:36.046418 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:36.046387 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ljvqz" Oct 30 12:45:37.351218 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:37.351175 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 12:45:43.331061 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:43.330967 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-596f5cc4c5-sgvjw"] Oct 30 12:45:45.503875 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:45.503819 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" podUID="d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5" containerName="registry" containerID="cri-o://ab6fff72e975fdabab14ac7131ecc78b6e8a72160c2d2768ba55327f238d71e9" gracePeriod=30 Oct 30 12:45:45.767655 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:45.767600 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:45:45.780848 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:45.780824 4931 generic.go:356] "Generic (PLEG): container finished" podID="d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5" containerID="ab6fff72e975fdabab14ac7131ecc78b6e8a72160c2d2768ba55327f238d71e9" exitCode=0 Oct 30 12:45:45.780966 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:45.780866 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" event={"ID":"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5","Type":"ContainerDied","Data":"ab6fff72e975fdabab14ac7131ecc78b6e8a72160c2d2768ba55327f238d71e9"} Oct 30 12:45:45.780966 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:45.780872 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" Oct 30 12:45:45.780966 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:45.780929 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bff985b5d-qwb5l" event={"ID":"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5","Type":"ContainerDied","Data":"788c8feb3a6101ec87781cf91e3f293764b1b54bb683b9d57e1f5aadfb00668f"} Oct 30 12:45:45.780966 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:45.780950 4931 scope.go:122] "RemoveContainer" containerID="ab6fff72e975fdabab14ac7131ecc78b6e8a72160c2d2768ba55327f238d71e9" Oct 30 12:45:45.788830 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:45.788808 4931 scope.go:122] "RemoveContainer" containerID="ab6fff72e975fdabab14ac7131ecc78b6e8a72160c2d2768ba55327f238d71e9" Oct 30 12:45:45.789154 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:45:45.789117 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab6fff72e975fdabab14ac7131ecc78b6e8a72160c2d2768ba55327f238d71e9\": container with ID starting with ab6fff72e975fdabab14ac7131ecc78b6e8a72160c2d2768ba55327f238d71e9 not found: ID does not exist" containerID="ab6fff72e975fdabab14ac7131ecc78b6e8a72160c2d2768ba55327f238d71e9" Oct 30 12:45:45.789231 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:45.789168 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab6fff72e975fdabab14ac7131ecc78b6e8a72160c2d2768ba55327f238d71e9"} err="failed to get container status \"ab6fff72e975fdabab14ac7131ecc78b6e8a72160c2d2768ba55327f238d71e9\": rpc error: code = NotFound desc = could not find container \"ab6fff72e975fdabab14ac7131ecc78b6e8a72160c2d2768ba55327f238d71e9\": container with ID starting with ab6fff72e975fdabab14ac7131ecc78b6e8a72160c2d2768ba55327f238d71e9 not found: ID does not exist" Oct 30 12:45:45.942784 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:45.942755 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-kube-api-access-jwpj2\" (UniqueName: \"kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-kube-api-access-jwpj2\") pod \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " Oct 30 12:45:45.942966 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:45.942799 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-certificates\") pod \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " Oct 30 12:45:45.942966 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:45.942843 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-trusted-ca\") pod \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " Oct 30 12:45:45.942966 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:45.942874 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-tls\" (UniqueName: \"kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-tls\") pod \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " Oct 30 12:45:45.942966 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:45.942929 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-installation-pull-secrets\") pod \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " Oct 30 12:45:45.942966 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:45.942957 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-ca-trust-extracted\") pod \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " Oct 30 12:45:45.943217 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:45.942986 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-bound-sa-token\") pod \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " Oct 30 12:45:45.943217 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:45.943021 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-image-registry-private-configuration\") pod \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\" (UID: \"d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5\") " Oct 30 12:45:45.943384 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:45.943353 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-trusted-ca" pod "d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5" (UID: "d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:45.943454 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:45.943417 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-certificates" pod "d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5" (UID: "d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:45.945561 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:45.945511 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-image-registry-private-configuration" pod "d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5" (UID: "d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:45:45.945695 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:45.945642 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-installation-pull-secrets" pod "d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5" (UID: "d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:45:45.945759 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:45.945705 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-tls" pod "d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5" (UID: "d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:45.945759 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:45.945739 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-kube-api-access-jwpj2" pod "d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5" (UID: "d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5"). InnerVolumeSpecName "kube-api-access-jwpj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:45.945867 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:45.945756 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-bound-sa-token" pod "d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5" (UID: "d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:45.949749 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:45.949714 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-ca-trust-extracted" pod "d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5" (UID: "d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:46.044411 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:46.044337 4931 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-installation-pull-secrets\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:45:46.044411 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:46.044371 4931 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-ca-trust-extracted\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:45:46.044411 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:46.044387 4931 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-bound-sa-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:45:46.044411 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:46.044402 4931 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-image-registry-private-configuration\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:45:46.044411 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:46.044417 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwpj2\" (UniqueName: \"kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-kube-api-access-jwpj2\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:45:46.044710 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:46.044431 4931 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-certificates\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:45:46.044710 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:46.044443 4931 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-trusted-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:45:46.044710 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:46.044456 4931 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5-registry-tls\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:45:46.132852 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:46.132819 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7bff985b5d-qwb5l"] Oct 30 12:45:46.145909 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:46.145885 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7bff985b5d-qwb5l"] Oct 30 12:45:47.510534 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:45:47.510497 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5" path="/var/lib/kubelet/pods/d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5/volumes" Oct 30 12:46:07.350633 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:46:07.350598 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 12:46:07.724375 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:07.724330 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="kube-system/konnectivity-agent-k2fjm" podUID="37620bdd-11a4-4c12-8be1-9813a85d575b" containerName="konnectivity-agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:46:07.724547 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:07.724467 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-k2fjm" Oct 30 12:46:07.725120 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:07.725093 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="kube-system/konnectivity-agent-k2fjm" podUID="37620bdd-11a4-4c12-8be1-9813a85d575b" containerName="konnectivity-agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:46:08.351158 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:08.351091 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" podUID="6000e43a-e175-46fe-b5f7-a417fc1ba5bb" containerName="registry" containerID="cri-o://468c8e9f73ad8f848c42905e9a4e32c481a9cbfde9a71cf4a329b0f543d311fd" gracePeriod=30 Oct 30 12:46:08.591742 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:08.591711 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:46:08.704580 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:08.704547 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-trusted-ca\") pod \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " Oct 30 12:46:08.704757 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:08.704610 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-ca-trust-extracted\") pod \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " Oct 30 12:46:08.704757 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:08.704641 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-tls\" (UniqueName: \"kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-tls\") pod \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " Oct 30 12:46:08.704757 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:08.704669 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-bound-sa-token\") pod \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " Oct 30 12:46:08.704757 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:08.704708 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-installation-pull-secrets\") pod \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " Oct 30 12:46:08.704757 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:08.704747 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-image-registry-private-configuration\") pod \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " Oct 30 12:46:08.705037 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:08.704808 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-kube-api-access-zks6k\" (UniqueName: \"kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-kube-api-access-zks6k\") pod \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " Oct 30 12:46:08.705037 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:08.704863 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-certificates\") pod \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\" (UID: \"6000e43a-e175-46fe-b5f7-a417fc1ba5bb\") " Oct 30 12:46:08.705137 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:08.705083 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-trusted-ca" pod "6000e43a-e175-46fe-b5f7-a417fc1ba5bb" (UID: "6000e43a-e175-46fe-b5f7-a417fc1ba5bb"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:46:08.705451 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:08.705423 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-certificates" pod "6000e43a-e175-46fe-b5f7-a417fc1ba5bb" (UID: "6000e43a-e175-46fe-b5f7-a417fc1ba5bb"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:46:08.707560 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:08.707516 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-installation-pull-secrets" pod "6000e43a-e175-46fe-b5f7-a417fc1ba5bb" (UID: "6000e43a-e175-46fe-b5f7-a417fc1ba5bb"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:46:08.707676 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:08.707591 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-image-registry-private-configuration" pod "6000e43a-e175-46fe-b5f7-a417fc1ba5bb" (UID: "6000e43a-e175-46fe-b5f7-a417fc1ba5bb"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:46:08.707996 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:08.707957 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-kube-api-access-zks6k" pod "6000e43a-e175-46fe-b5f7-a417fc1ba5bb" (UID: "6000e43a-e175-46fe-b5f7-a417fc1ba5bb"). InnerVolumeSpecName "kube-api-access-zks6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:46:08.708381 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:08.708350 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-bound-sa-token" pod "6000e43a-e175-46fe-b5f7-a417fc1ba5bb" (UID: "6000e43a-e175-46fe-b5f7-a417fc1ba5bb"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:46:08.708491 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:08.708417 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-tls" pod "6000e43a-e175-46fe-b5f7-a417fc1ba5bb" (UID: "6000e43a-e175-46fe-b5f7-a417fc1ba5bb"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:46:08.713297 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:08.713255 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-ca-trust-extracted" pod "6000e43a-e175-46fe-b5f7-a417fc1ba5bb" (UID: "6000e43a-e175-46fe-b5f7-a417fc1ba5bb"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:46:08.806382 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:08.806340 4931 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-installation-pull-secrets\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:46:08.806382 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:08.806379 4931 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-image-registry-private-configuration\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:46:08.806382 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:08.806391 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zks6k\" (UniqueName: \"kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-kube-api-access-zks6k\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:46:08.806597 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:08.806400 4931 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-certificates\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:46:08.806597 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:08.806410 4931 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-trusted-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:46:08.806597 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:08.806419 4931 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-ca-trust-extracted\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:46:08.806597 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:08.806427 4931 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-registry-tls\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:46:08.806597 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:08.806435 4931 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6000e43a-e175-46fe-b5f7-a417fc1ba5bb-bound-sa-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:46:08.871115 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:08.871076 4931 generic.go:356] "Generic (PLEG): container finished" podID="6000e43a-e175-46fe-b5f7-a417fc1ba5bb" containerID="468c8e9f73ad8f848c42905e9a4e32c481a9cbfde9a71cf4a329b0f543d311fd" exitCode=0 Oct 30 12:46:08.871240 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:08.871124 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" Oct 30 12:46:08.871240 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:08.871130 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" event={"ID":"6000e43a-e175-46fe-b5f7-a417fc1ba5bb","Type":"ContainerDied","Data":"468c8e9f73ad8f848c42905e9a4e32c481a9cbfde9a71cf4a329b0f543d311fd"} Oct 30 12:46:08.871240 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:08.871175 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-596f5cc4c5-sgvjw" event={"ID":"6000e43a-e175-46fe-b5f7-a417fc1ba5bb","Type":"ContainerDied","Data":"93c9e8970612a0952d3376de288d80ab27f9eb95263ff91dc918c2fd8e598ef3"} Oct 30 12:46:08.871240 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:08.871198 4931 scope.go:122] "RemoveContainer" containerID="468c8e9f73ad8f848c42905e9a4e32c481a9cbfde9a71cf4a329b0f543d311fd" Oct 30 12:46:08.879774 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:08.879751 4931 scope.go:122] "RemoveContainer" containerID="468c8e9f73ad8f848c42905e9a4e32c481a9cbfde9a71cf4a329b0f543d311fd" Oct 30 12:46:08.880039 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:46:08.880016 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"468c8e9f73ad8f848c42905e9a4e32c481a9cbfde9a71cf4a329b0f543d311fd\": container with ID starting with 468c8e9f73ad8f848c42905e9a4e32c481a9cbfde9a71cf4a329b0f543d311fd not found: ID does not exist" containerID="468c8e9f73ad8f848c42905e9a4e32c481a9cbfde9a71cf4a329b0f543d311fd" Oct 30 12:46:08.880086 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:08.880047 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"468c8e9f73ad8f848c42905e9a4e32c481a9cbfde9a71cf4a329b0f543d311fd"} err="failed to get container status \"468c8e9f73ad8f848c42905e9a4e32c481a9cbfde9a71cf4a329b0f543d311fd\": rpc error: code = NotFound desc = could not find container \"468c8e9f73ad8f848c42905e9a4e32c481a9cbfde9a71cf4a329b0f543d311fd\": container with ID starting with 468c8e9f73ad8f848c42905e9a4e32c481a9cbfde9a71cf4a329b0f543d311fd not found: ID does not exist" Oct 30 12:46:08.930687 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:08.930661 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-596f5cc4c5-sgvjw"] Oct 30 12:46:08.944327 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:08.944305 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-596f5cc4c5-sgvjw"] Oct 30 12:46:09.510813 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:09.510783 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6000e43a-e175-46fe-b5f7-a417fc1ba5bb" path="/var/lib/kubelet/pods/6000e43a-e175-46fe-b5f7-a417fc1ba5bb/volumes" Oct 30 12:46:11.883093 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:11.883058 4931 generic.go:356] "Generic (PLEG): container finished" podID="73b3e3a7-93fa-49c2-a8f1-4188a22b99a5" containerID="6c3473393491490b7e955cf33f49973c0223af60dfae4f6463a70d23709cb6ac" exitCode=0 Oct 30 12:46:11.883456 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:11.883133 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-sztw6" event={"ID":"73b3e3a7-93fa-49c2-a8f1-4188a22b99a5","Type":"ContainerDied","Data":"6c3473393491490b7e955cf33f49973c0223af60dfae4f6463a70d23709cb6ac"} Oct 30 12:46:11.883575 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:11.883557 4931 scope.go:122] "RemoveContainer" containerID="6c3473393491490b7e955cf33f49973c0223af60dfae4f6463a70d23709cb6ac" Oct 30 12:46:12.888487 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:12.888456 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-sztw6" event={"ID":"73b3e3a7-93fa-49c2-a8f1-4188a22b99a5","Type":"ContainerStarted","Data":"61f7b14beb1425ed176b2c593edfd9cf006e3d17502e874b09f6456881e1eaf8"} Oct 30 12:46:17.911832 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:17.911791 4931 generic.go:356] "Generic (PLEG): container finished" podID="e4bf0a41-a4bd-47ad-93b1-088500499acb" containerID="0c4d37ba87c40a37fe74f5eb9c86238d481fb22536037132f94335bb34ce6983" exitCode=0 Oct 30 12:46:17.912230 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:17.911847 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzhq7" event={"ID":"e4bf0a41-a4bd-47ad-93b1-088500499acb","Type":"ContainerDied","Data":"0c4d37ba87c40a37fe74f5eb9c86238d481fb22536037132f94335bb34ce6983"} Oct 30 12:46:17.912230 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:17.912175 4931 scope.go:122] "RemoveContainer" containerID="0c4d37ba87c40a37fe74f5eb9c86238d481fb22536037132f94335bb34ce6983" Oct 30 12:46:18.916721 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:18.916689 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzhq7" event={"ID":"e4bf0a41-a4bd-47ad-93b1-088500499acb","Type":"ContainerStarted","Data":"55ad9e4f625d7cc313464a3917d879858452026e661b87de3ae8de0705f3cb7a"} Oct 30 12:46:36.981458 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:36.981420 4931 generic.go:356] "Generic (PLEG): container finished" podID="71e82393-029b-4388-86df-8bd068720867" containerID="6de6c456fb82b0d9abcd886b97ed82eacd6844d60d93bac83bc5ba18286d2f7b" exitCode=0 Oct 30 12:46:36.981845 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:36.981498 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pptmg" event={"ID":"71e82393-029b-4388-86df-8bd068720867","Type":"ContainerDied","Data":"6de6c456fb82b0d9abcd886b97ed82eacd6844d60d93bac83bc5ba18286d2f7b"} Oct 30 12:46:36.981845 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:36.981827 4931 scope.go:122] "RemoveContainer" containerID="6de6c456fb82b0d9abcd886b97ed82eacd6844d60d93bac83bc5ba18286d2f7b" Oct 30 12:46:37.351366 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:46:37.351289 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 12:46:37.726142 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:37.726117 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-k2fjm" Oct 30 12:46:37.986908 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:46:37.986823 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pptmg" event={"ID":"71e82393-029b-4388-86df-8bd068720867","Type":"ContainerStarted","Data":"6d343cdb4777b667bdb8012d044a4eda894cad4128f5616e8399353822800db9"} Oct 30 12:47:07.350862 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:47:07.350822 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 12:47:37.350341 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:47:37.350300 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 12:48:07.350421 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:48:07.350379 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 12:48:37.350532 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:48:37.350486 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 12:49:07.351185 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:49:07.351146 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 12:49:37.350732 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:49:37.350692 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 12:49:55.582360 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:49:55.582327 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hrk8"] Oct 30 12:49:55.582813 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:49:55.582606 4931 cpu_manager.go:414] "containerMap: removing container" podUID="6000e43a-e175-46fe-b5f7-a417fc1ba5bb" containerName="registry" Oct 30 12:49:55.582813 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:49:55.582617 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6000e43a-e175-46fe-b5f7-a417fc1ba5bb" containerName="registry" Oct 30 12:49:55.582813 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:49:55.582627 4931 cpu_manager.go:414] "containerMap: removing container" podUID="d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5" containerName="registry" Oct 30 12:49:55.582813 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:49:55.582632 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5" containerName="registry" Oct 30 12:49:55.582813 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:49:55.582693 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6000e43a-e175-46fe-b5f7-a417fc1ba5bb" containerName="registry" Oct 30 12:49:55.582813 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:49:55.582699 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8d7b93d-1b9a-4c2e-b80b-ddeb336f48a5" containerName="registry" Oct 30 12:49:55.585678 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:49:55.585661 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hrk8" Oct 30 12:49:55.596069 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:49:55.596040 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab81d68f-b3b5-42ff-b16b-34877e1d51ec-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hrk8\" (UID: \"ab81d68f-b3b5-42ff-b16b-34877e1d51ec\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hrk8" Oct 30 12:49:55.596171 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:49:55.596136 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab81d68f-b3b5-42ff-b16b-34877e1d51ec-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hrk8\" (UID: \"ab81d68f-b3b5-42ff-b16b-34877e1d51ec\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hrk8" Oct 30 12:49:55.596217 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:49:55.596171 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtw7q\" (UniqueName: \"kubernetes.io/projected/ab81d68f-b3b5-42ff-b16b-34877e1d51ec-kube-api-access-wtw7q\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hrk8\" (UID: \"ab81d68f-b3b5-42ff-b16b-34877e1d51ec\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hrk8" Oct 30 12:49:55.616819 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:49:55.616791 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 12:49:55.616969 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:49:55.616835 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-6nwr5\"" Oct 30 12:49:55.616969 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:49:55.616840 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 12:49:55.618038 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:49:55.618016 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hrk8"] Oct 30 12:49:55.697001 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:49:55.696972 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab81d68f-b3b5-42ff-b16b-34877e1d51ec-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hrk8\" (UID: \"ab81d68f-b3b5-42ff-b16b-34877e1d51ec\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hrk8" Oct 30 12:49:55.697113 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:49:55.697014 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab81d68f-b3b5-42ff-b16b-34877e1d51ec-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hrk8\" (UID: \"ab81d68f-b3b5-42ff-b16b-34877e1d51ec\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hrk8" Oct 30 12:49:55.697113 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:49:55.697036 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtw7q\" (UniqueName: \"kubernetes.io/projected/ab81d68f-b3b5-42ff-b16b-34877e1d51ec-kube-api-access-wtw7q\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hrk8\" (UID: \"ab81d68f-b3b5-42ff-b16b-34877e1d51ec\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hrk8" Oct 30 12:49:55.697341 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:49:55.697322 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab81d68f-b3b5-42ff-b16b-34877e1d51ec-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hrk8\" (UID: \"ab81d68f-b3b5-42ff-b16b-34877e1d51ec\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hrk8" Oct 30 12:49:55.697395 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:49:55.697359 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab81d68f-b3b5-42ff-b16b-34877e1d51ec-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hrk8\" (UID: \"ab81d68f-b3b5-42ff-b16b-34877e1d51ec\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hrk8" Oct 30 12:49:55.728751 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:49:55.728723 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtw7q\" (UniqueName: \"kubernetes.io/projected/ab81d68f-b3b5-42ff-b16b-34877e1d51ec-kube-api-access-wtw7q\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hrk8\" (UID: \"ab81d68f-b3b5-42ff-b16b-34877e1d51ec\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hrk8" Oct 30 12:49:55.901203 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:49:55.901126 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hrk8" Oct 30 12:49:56.204646 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:49:56.204612 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hrk8"] Oct 30 12:49:56.207799 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:49:56.207772 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab81d68f_b3b5_42ff_b16b_34877e1d51ec.slice/crio-a42c7f3d40cea570d9cb358fde07d0e947bab5695d6023feed66818b66243dd0 WatchSource:0}: Error finding container a42c7f3d40cea570d9cb358fde07d0e947bab5695d6023feed66818b66243dd0: Status 404 returned error can't find the container with id a42c7f3d40cea570d9cb358fde07d0e947bab5695d6023feed66818b66243dd0 Oct 30 12:49:56.668091 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:49:56.668009 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hrk8" event={"ID":"ab81d68f-b3b5-42ff-b16b-34877e1d51ec","Type":"ContainerStarted","Data":"a42c7f3d40cea570d9cb358fde07d0e947bab5695d6023feed66818b66243dd0"} Oct 30 12:50:01.523000 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:01.522971 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/2.log" Oct 30 12:50:01.524969 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:01.524947 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/2.log" Oct 30 12:50:01.688429 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:01.688348 4931 generic.go:356] "Generic (PLEG): container finished" podID="ab81d68f-b3b5-42ff-b16b-34877e1d51ec" containerID="72b68c754a06b0e4c87852eca6526a72ae9d45ef99fd7e1883dc8c4684729f8d" exitCode=0 Oct 30 12:50:01.688577 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:01.688431 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hrk8" event={"ID":"ab81d68f-b3b5-42ff-b16b-34877e1d51ec","Type":"ContainerDied","Data":"72b68c754a06b0e4c87852eca6526a72ae9d45ef99fd7e1883dc8c4684729f8d"} Oct 30 12:50:04.704461 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:04.704426 4931 generic.go:356] "Generic (PLEG): container finished" podID="ab81d68f-b3b5-42ff-b16b-34877e1d51ec" containerID="495148f5311a6686f07e34937c635730329427d3a71e0af53bf5f4c493af862c" exitCode=0 Oct 30 12:50:04.704855 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:04.704507 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hrk8" event={"ID":"ab81d68f-b3b5-42ff-b16b-34877e1d51ec","Type":"ContainerDied","Data":"495148f5311a6686f07e34937c635730329427d3a71e0af53bf5f4c493af862c"} Oct 30 12:50:04.705708 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:04.705689 4931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:50:07.356766 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:50:07.356724 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 12:50:10.729813 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:10.729779 4931 generic.go:356] "Generic (PLEG): container finished" podID="ab81d68f-b3b5-42ff-b16b-34877e1d51ec" containerID="46bc7d78ce1f852883ae083380e6dc431d27ab18b1c9ef55a145db87b4b837d7" exitCode=0 Oct 30 12:50:10.730205 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:10.729860 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hrk8" event={"ID":"ab81d68f-b3b5-42ff-b16b-34877e1d51ec","Type":"ContainerDied","Data":"46bc7d78ce1f852883ae083380e6dc431d27ab18b1c9ef55a145db87b4b837d7"} Oct 30 12:50:11.862053 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:11.862026 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hrk8" Oct 30 12:50:11.914491 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:11.914460 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab81d68f-b3b5-42ff-b16b-34877e1d51ec-util\" (UniqueName: \"kubernetes.io/empty-dir/ab81d68f-b3b5-42ff-b16b-34877e1d51ec-util\") pod \"ab81d68f-b3b5-42ff-b16b-34877e1d51ec\" (UID: \"ab81d68f-b3b5-42ff-b16b-34877e1d51ec\") " Oct 30 12:50:11.914663 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:11.914505 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab81d68f-b3b5-42ff-b16b-34877e1d51ec-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab81d68f-b3b5-42ff-b16b-34877e1d51ec-bundle\") pod \"ab81d68f-b3b5-42ff-b16b-34877e1d51ec\" (UID: \"ab81d68f-b3b5-42ff-b16b-34877e1d51ec\") " Oct 30 12:50:11.914663 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:11.914558 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab81d68f-b3b5-42ff-b16b-34877e1d51ec-kube-api-access-wtw7q\" (UniqueName: \"kubernetes.io/projected/ab81d68f-b3b5-42ff-b16b-34877e1d51ec-kube-api-access-wtw7q\") pod \"ab81d68f-b3b5-42ff-b16b-34877e1d51ec\" (UID: \"ab81d68f-b3b5-42ff-b16b-34877e1d51ec\") " Oct 30 12:50:11.915315 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:11.915262 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab81d68f-b3b5-42ff-b16b-34877e1d51ec-bundle" pod "ab81d68f-b3b5-42ff-b16b-34877e1d51ec" (UID: "ab81d68f-b3b5-42ff-b16b-34877e1d51ec"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:50:11.916945 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:11.916912 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab81d68f-b3b5-42ff-b16b-34877e1d51ec-kube-api-access-wtw7q" pod "ab81d68f-b3b5-42ff-b16b-34877e1d51ec" (UID: "ab81d68f-b3b5-42ff-b16b-34877e1d51ec"). InnerVolumeSpecName "kube-api-access-wtw7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:50:11.918890 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:11.918865 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab81d68f-b3b5-42ff-b16b-34877e1d51ec-util" pod "ab81d68f-b3b5-42ff-b16b-34877e1d51ec" (UID: "ab81d68f-b3b5-42ff-b16b-34877e1d51ec"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:50:12.015441 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:12.015358 4931 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab81d68f-b3b5-42ff-b16b-34877e1d51ec-util\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:50:12.015441 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:12.015388 4931 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab81d68f-b3b5-42ff-b16b-34877e1d51ec-bundle\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:50:12.015441 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:12.015398 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtw7q\" (UniqueName: \"kubernetes.io/projected/ab81d68f-b3b5-42ff-b16b-34877e1d51ec-kube-api-access-wtw7q\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:50:12.739068 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:12.739024 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hrk8" event={"ID":"ab81d68f-b3b5-42ff-b16b-34877e1d51ec","Type":"ContainerDied","Data":"a42c7f3d40cea570d9cb358fde07d0e947bab5695d6023feed66818b66243dd0"} Oct 30 12:50:12.739068 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:12.739065 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hrk8" Oct 30 12:50:12.739305 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:12.739083 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a42c7f3d40cea570d9cb358fde07d0e947bab5695d6023feed66818b66243dd0" Oct 30 12:50:19.298066 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:19.298020 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-c47t7"] Oct 30 12:50:19.298596 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:19.298467 4931 cpu_manager.go:414] "containerMap: removing container" podUID="ab81d68f-b3b5-42ff-b16b-34877e1d51ec" containerName="extract" Oct 30 12:50:19.298596 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:19.298486 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab81d68f-b3b5-42ff-b16b-34877e1d51ec" containerName="extract" Oct 30 12:50:19.298596 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:19.298511 4931 cpu_manager.go:414] "containerMap: removing container" podUID="ab81d68f-b3b5-42ff-b16b-34877e1d51ec" containerName="pull" Oct 30 12:50:19.298596 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:19.298519 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab81d68f-b3b5-42ff-b16b-34877e1d51ec" containerName="pull" Oct 30 12:50:19.298596 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:19.298554 4931 cpu_manager.go:414] "containerMap: removing container" podUID="ab81d68f-b3b5-42ff-b16b-34877e1d51ec" containerName="util" Oct 30 12:50:19.298596 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:19.298560 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab81d68f-b3b5-42ff-b16b-34877e1d51ec" containerName="util" Oct 30 12:50:19.298888 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:19.298659 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab81d68f-b3b5-42ff-b16b-34877e1d51ec" containerName="extract" Oct 30 12:50:19.301708 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:19.301688 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-c47t7" Oct 30 12:50:19.339320 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:19.339298 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-vkghc\"" Oct 30 12:50:19.351725 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:19.351692 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 12:50:19.351725 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:19.351694 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:50:19.353358 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:19.353338 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-c47t7"] Oct 30 12:50:19.374342 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:19.374304 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a96a37c-b02f-46a4-98b8-c851ff4fbc79-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-c47t7\" (UID: \"4a96a37c-b02f-46a4-98b8-c851ff4fbc79\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-c47t7" Oct 30 12:50:19.374701 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:19.374679 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nphvr\" (UniqueName: \"kubernetes.io/projected/4a96a37c-b02f-46a4-98b8-c851ff4fbc79-kube-api-access-nphvr\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-c47t7\" (UID: \"4a96a37c-b02f-46a4-98b8-c851ff4fbc79\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-c47t7" Oct 30 12:50:19.475988 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:19.475944 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nphvr\" (UniqueName: \"kubernetes.io/projected/4a96a37c-b02f-46a4-98b8-c851ff4fbc79-kube-api-access-nphvr\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-c47t7\" (UID: \"4a96a37c-b02f-46a4-98b8-c851ff4fbc79\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-c47t7" Oct 30 12:50:19.476153 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:19.476012 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a96a37c-b02f-46a4-98b8-c851ff4fbc79-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-c47t7\" (UID: \"4a96a37c-b02f-46a4-98b8-c851ff4fbc79\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-c47t7" Oct 30 12:50:19.476404 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:19.476386 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a96a37c-b02f-46a4-98b8-c851ff4fbc79-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-c47t7\" (UID: \"4a96a37c-b02f-46a4-98b8-c851ff4fbc79\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-c47t7" Oct 30 12:50:19.504055 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:19.504016 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nphvr\" (UniqueName: \"kubernetes.io/projected/4a96a37c-b02f-46a4-98b8-c851ff4fbc79-kube-api-access-nphvr\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-c47t7\" (UID: \"4a96a37c-b02f-46a4-98b8-c851ff4fbc79\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-c47t7" Oct 30 12:50:19.613122 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:19.613024 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-c47t7" Oct 30 12:50:19.958332 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:19.958305 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-c47t7"] Oct 30 12:50:19.961305 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:50:19.961258 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a96a37c_b02f_46a4_98b8_c851ff4fbc79.slice/crio-bd19e1d9d7a4b8935481d941198a3171d1795f9c62a1c34495f4d20689782605 WatchSource:0}: Error finding container bd19e1d9d7a4b8935481d941198a3171d1795f9c62a1c34495f4d20689782605: Status 404 returned error can't find the container with id bd19e1d9d7a4b8935481d941198a3171d1795f9c62a1c34495f4d20689782605 Oct 30 12:50:20.769015 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:20.768971 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-c47t7" event={"ID":"4a96a37c-b02f-46a4-98b8-c851ff4fbc79","Type":"ContainerStarted","Data":"bd19e1d9d7a4b8935481d941198a3171d1795f9c62a1c34495f4d20689782605"} Oct 30 12:50:21.774957 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:21.774919 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-c47t7" event={"ID":"4a96a37c-b02f-46a4-98b8-c851ff4fbc79","Type":"ContainerStarted","Data":"67b0dd03b504e2b5b5c3d97a9ac1bf585ee929a3dca9145aae73a4f62ec9caf4"} Oct 30 12:50:21.856951 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:21.856886 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-c47t7" podStartSLOduration=1.459698585 podStartE2EDuration="2.856867401s" totalImagesPullingTime="1.397168816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:50:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:50:21.85613107 +0000 UTC m=+320.459629515" watchObservedRunningTime="2026-10-30 12:50:21.856867401 +0000 UTC m=+320.460365848" Oct 30 12:50:23.609224 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:23.609181 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc854nrkp"] Oct 30 12:50:23.613116 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:23.613091 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc854nrkp" Oct 30 12:50:23.630800 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:23.630775 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"61b1ff68c7255ee5c5329fe19a5d96293a9578-ce91a85d-dockercfg-xkpsd\"" Oct 30 12:50:23.632246 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:23.632230 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 12:50:23.645859 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:23.645835 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc854nrkp"] Oct 30 12:50:23.714825 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:23.714787 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00e5faf2-ad90-4913-b4a4-a9b699eec0b1-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc854nrkp\" (UID: \"00e5faf2-ad90-4913-b4a4-a9b699eec0b1\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc854nrkp" Oct 30 12:50:23.714825 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:23.714825 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00e5faf2-ad90-4913-b4a4-a9b699eec0b1-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc854nrkp\" (UID: \"00e5faf2-ad90-4913-b4a4-a9b699eec0b1\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc854nrkp" Oct 30 12:50:23.715036 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:23.714925 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzb2q\" (UniqueName: \"kubernetes.io/projected/00e5faf2-ad90-4913-b4a4-a9b699eec0b1-kube-api-access-wzb2q\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc854nrkp\" (UID: \"00e5faf2-ad90-4913-b4a4-a9b699eec0b1\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc854nrkp" Oct 30 12:50:23.737899 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:23.737873 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 12:50:23.815667 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:23.815633 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00e5faf2-ad90-4913-b4a4-a9b699eec0b1-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc854nrkp\" (UID: \"00e5faf2-ad90-4913-b4a4-a9b699eec0b1\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc854nrkp" Oct 30 12:50:23.815667 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:23.815665 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00e5faf2-ad90-4913-b4a4-a9b699eec0b1-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc854nrkp\" (UID: \"00e5faf2-ad90-4913-b4a4-a9b699eec0b1\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc854nrkp" Oct 30 12:50:23.815973 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:23.815696 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzb2q\" (UniqueName: \"kubernetes.io/projected/00e5faf2-ad90-4913-b4a4-a9b699eec0b1-kube-api-access-wzb2q\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc854nrkp\" (UID: \"00e5faf2-ad90-4913-b4a4-a9b699eec0b1\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc854nrkp" Oct 30 12:50:23.816045 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:23.816030 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00e5faf2-ad90-4913-b4a4-a9b699eec0b1-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc854nrkp\" (UID: \"00e5faf2-ad90-4913-b4a4-a9b699eec0b1\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc854nrkp" Oct 30 12:50:23.816111 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:23.816093 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00e5faf2-ad90-4913-b4a4-a9b699eec0b1-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc854nrkp\" (UID: \"00e5faf2-ad90-4913-b4a4-a9b699eec0b1\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc854nrkp" Oct 30 12:50:23.846812 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:23.846781 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzb2q\" (UniqueName: \"kubernetes.io/projected/00e5faf2-ad90-4913-b4a4-a9b699eec0b1-kube-api-access-wzb2q\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc854nrkp\" (UID: \"00e5faf2-ad90-4913-b4a4-a9b699eec0b1\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc854nrkp" Oct 30 12:50:23.928974 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:23.928878 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc854nrkp" Oct 30 12:50:24.227655 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:24.227631 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc854nrkp"] Oct 30 12:50:24.229878 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:50:24.229851 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00e5faf2_ad90_4913_b4a4_a9b699eec0b1.slice/crio-e6aceaa96dc4f22c308bf4f648f452a64b1ddfc42ef16a10bc0c6ad46a18da78 WatchSource:0}: Error finding container e6aceaa96dc4f22c308bf4f648f452a64b1ddfc42ef16a10bc0c6ad46a18da78: Status 404 returned error can't find the container with id e6aceaa96dc4f22c308bf4f648f452a64b1ddfc42ef16a10bc0c6ad46a18da78 Oct 30 12:50:24.793519 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:24.793485 4931 generic.go:356] "Generic (PLEG): container finished" podID="00e5faf2-ad90-4913-b4a4-a9b699eec0b1" containerID="c54c9428434d49d1064bb1719dd6b1cf0db06e7d7d4c3e3e7d46772bd62e2243" exitCode=0 Oct 30 12:50:24.793995 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:24.793566 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc854nrkp" event={"ID":"00e5faf2-ad90-4913-b4a4-a9b699eec0b1","Type":"ContainerDied","Data":"c54c9428434d49d1064bb1719dd6b1cf0db06e7d7d4c3e3e7d46772bd62e2243"} Oct 30 12:50:24.793995 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:24.793642 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc854nrkp" event={"ID":"00e5faf2-ad90-4913-b4a4-a9b699eec0b1","Type":"ContainerStarted","Data":"e6aceaa96dc4f22c308bf4f648f452a64b1ddfc42ef16a10bc0c6ad46a18da78"} Oct 30 12:50:24.998073 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:24.998040 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-qq9cd"] Oct 30 12:50:25.001052 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:25.001027 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qq9cd" Oct 30 12:50:25.021565 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:25.021538 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-qq9cd"] Oct 30 12:50:25.023135 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:25.023111 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 12:50:25.023252 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:25.023236 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 12:50:25.023700 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:25.023677 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-qj92x\"" Oct 30 12:50:25.126804 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:25.126725 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b722ad3-7067-4329-9bd0-96345c9b6826-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-qq9cd\" (UID: \"8b722ad3-7067-4329-9bd0-96345c9b6826\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qq9cd" Oct 30 12:50:25.126804 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:25.126762 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rwr7\" (UniqueName: \"kubernetes.io/projected/8b722ad3-7067-4329-9bd0-96345c9b6826-kube-api-access-7rwr7\") pod \"cert-manager-cainjector-5c57bbd4fc-qq9cd\" (UID: \"8b722ad3-7067-4329-9bd0-96345c9b6826\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qq9cd" Oct 30 12:50:25.227642 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:25.227603 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b722ad3-7067-4329-9bd0-96345c9b6826-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-qq9cd\" (UID: \"8b722ad3-7067-4329-9bd0-96345c9b6826\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qq9cd" Oct 30 12:50:25.227642 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:25.227648 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rwr7\" (UniqueName: \"kubernetes.io/projected/8b722ad3-7067-4329-9bd0-96345c9b6826-kube-api-access-7rwr7\") pod \"cert-manager-cainjector-5c57bbd4fc-qq9cd\" (UID: \"8b722ad3-7067-4329-9bd0-96345c9b6826\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qq9cd" Oct 30 12:50:25.258852 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:25.258817 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rwr7\" (UniqueName: \"kubernetes.io/projected/8b722ad3-7067-4329-9bd0-96345c9b6826-kube-api-access-7rwr7\") pod \"cert-manager-cainjector-5c57bbd4fc-qq9cd\" (UID: \"8b722ad3-7067-4329-9bd0-96345c9b6826\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qq9cd" Oct 30 12:50:25.259969 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:25.259950 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b722ad3-7067-4329-9bd0-96345c9b6826-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-qq9cd\" (UID: \"8b722ad3-7067-4329-9bd0-96345c9b6826\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qq9cd" Oct 30 12:50:25.325074 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:25.325035 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qq9cd" Oct 30 12:50:25.653205 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:25.653176 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-qq9cd"] Oct 30 12:50:25.654368 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:50:25.654337 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b722ad3_7067_4329_9bd0_96345c9b6826.slice/crio-c65a7a433bc644cd0e39956eb1b2ce849d4e26ad59504614c9893179ada2ce4a WatchSource:0}: Error finding container c65a7a433bc644cd0e39956eb1b2ce849d4e26ad59504614c9893179ada2ce4a: Status 404 returned error can't find the container with id c65a7a433bc644cd0e39956eb1b2ce849d4e26ad59504614c9893179ada2ce4a Oct 30 12:50:25.799702 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:25.799658 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qq9cd" event={"ID":"8b722ad3-7067-4329-9bd0-96345c9b6826","Type":"ContainerStarted","Data":"c65a7a433bc644cd0e39956eb1b2ce849d4e26ad59504614c9893179ada2ce4a"} Oct 30 12:50:27.812757 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:27.812709 4931 generic.go:356] "Generic (PLEG): container finished" podID="00e5faf2-ad90-4913-b4a4-a9b699eec0b1" containerID="dd94a1e3b667d5e5f8280fa8bf4f8e58908f88926826fd078d44a7b9ca03d611" exitCode=0 Oct 30 12:50:27.813279 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:27.812796 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc854nrkp" event={"ID":"00e5faf2-ad90-4913-b4a4-a9b699eec0b1","Type":"ContainerDied","Data":"dd94a1e3b667d5e5f8280fa8bf4f8e58908f88926826fd078d44a7b9ca03d611"} Oct 30 12:50:28.818616 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:28.818582 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qq9cd" event={"ID":"8b722ad3-7067-4329-9bd0-96345c9b6826","Type":"ContainerStarted","Data":"f6e6ac4000aa6a0d263ab5d1bef03eabad8f4068533fa0184a3ca2ab02a549ba"} Oct 30 12:50:28.820691 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:28.820656 4931 generic.go:356] "Generic (PLEG): container finished" podID="00e5faf2-ad90-4913-b4a4-a9b699eec0b1" containerID="934a1dba7dd76fd36fb1948d8d686af3bb71b5baecab84aed26d4391293401fb" exitCode=0 Oct 30 12:50:28.820820 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:28.820726 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc854nrkp" event={"ID":"00e5faf2-ad90-4913-b4a4-a9b699eec0b1","Type":"ContainerDied","Data":"934a1dba7dd76fd36fb1948d8d686af3bb71b5baecab84aed26d4391293401fb"} Oct 30 12:50:28.865741 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:28.865676 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qq9cd" podStartSLOduration=2.403633693 podStartE2EDuration="4.86566251s" totalImagesPullingTime="2.462028817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:50:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:50:28.864211308 +0000 UTC m=+327.467709751" watchObservedRunningTime="2026-10-30 12:50:28.86566251 +0000 UTC m=+327.469160955" Oct 30 12:50:29.945799 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:29.945764 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc854nrkp" Oct 30 12:50:30.071244 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:30.071209 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00e5faf2-ad90-4913-b4a4-a9b699eec0b1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/00e5faf2-ad90-4913-b4a4-a9b699eec0b1-bundle\") pod \"00e5faf2-ad90-4913-b4a4-a9b699eec0b1\" (UID: \"00e5faf2-ad90-4913-b4a4-a9b699eec0b1\") " Oct 30 12:50:30.071467 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:30.071261 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00e5faf2-ad90-4913-b4a4-a9b699eec0b1-util\" (UniqueName: \"kubernetes.io/empty-dir/00e5faf2-ad90-4913-b4a4-a9b699eec0b1-util\") pod \"00e5faf2-ad90-4913-b4a4-a9b699eec0b1\" (UID: \"00e5faf2-ad90-4913-b4a4-a9b699eec0b1\") " Oct 30 12:50:30.071467 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:30.071356 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00e5faf2-ad90-4913-b4a4-a9b699eec0b1-kube-api-access-wzb2q\" (UniqueName: \"kubernetes.io/projected/00e5faf2-ad90-4913-b4a4-a9b699eec0b1-kube-api-access-wzb2q\") pod \"00e5faf2-ad90-4913-b4a4-a9b699eec0b1\" (UID: \"00e5faf2-ad90-4913-b4a4-a9b699eec0b1\") " Oct 30 12:50:30.071702 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:30.071662 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e5faf2-ad90-4913-b4a4-a9b699eec0b1-bundle" pod "00e5faf2-ad90-4913-b4a4-a9b699eec0b1" (UID: "00e5faf2-ad90-4913-b4a4-a9b699eec0b1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:50:30.076288 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:30.076187 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00e5faf2-ad90-4913-b4a4-a9b699eec0b1-kube-api-access-wzb2q" pod "00e5faf2-ad90-4913-b4a4-a9b699eec0b1" (UID: "00e5faf2-ad90-4913-b4a4-a9b699eec0b1"). InnerVolumeSpecName "kube-api-access-wzb2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:50:30.077678 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:30.077651 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e5faf2-ad90-4913-b4a4-a9b699eec0b1-util" pod "00e5faf2-ad90-4913-b4a4-a9b699eec0b1" (UID: "00e5faf2-ad90-4913-b4a4-a9b699eec0b1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:50:30.172761 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:30.172712 4931 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00e5faf2-ad90-4913-b4a4-a9b699eec0b1-bundle\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:50:30.172761 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:30.172752 4931 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00e5faf2-ad90-4913-b4a4-a9b699eec0b1-util\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:50:30.172761 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:30.172766 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzb2q\" (UniqueName: \"kubernetes.io/projected/00e5faf2-ad90-4913-b4a4-a9b699eec0b1-kube-api-access-wzb2q\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:50:30.829964 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:30.829911 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc854nrkp" event={"ID":"00e5faf2-ad90-4913-b4a4-a9b699eec0b1","Type":"ContainerDied","Data":"e6aceaa96dc4f22c308bf4f648f452a64b1ddfc42ef16a10bc0c6ad46a18da78"} Oct 30 12:50:30.829964 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:30.829962 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc854nrkp" Oct 30 12:50:30.830179 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:30.829983 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6aceaa96dc4f22c308bf4f648f452a64b1ddfc42ef16a10bc0c6ad46a18da78" Oct 30 12:50:37.351772 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:50:37.351733 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 12:50:42.474668 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:42.474635 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bk7xg"] Oct 30 12:50:42.475048 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:42.474964 4931 cpu_manager.go:414] "containerMap: removing container" podUID="00e5faf2-ad90-4913-b4a4-a9b699eec0b1" containerName="util" Oct 30 12:50:42.475048 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:42.474976 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00e5faf2-ad90-4913-b4a4-a9b699eec0b1" containerName="util" Oct 30 12:50:42.475048 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:42.475003 4931 cpu_manager.go:414] "containerMap: removing container" podUID="00e5faf2-ad90-4913-b4a4-a9b699eec0b1" containerName="pull" Oct 30 12:50:42.475048 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:42.475008 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00e5faf2-ad90-4913-b4a4-a9b699eec0b1" containerName="pull" Oct 30 12:50:42.475048 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:42.475040 4931 cpu_manager.go:414] "containerMap: removing container" podUID="00e5faf2-ad90-4913-b4a4-a9b699eec0b1" containerName="extract" Oct 30 12:50:42.475048 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:42.475045 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00e5faf2-ad90-4913-b4a4-a9b699eec0b1" containerName="extract" Oct 30 12:50:42.475244 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:42.475101 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="00e5faf2-ad90-4913-b4a4-a9b699eec0b1" containerName="extract" Oct 30 12:50:42.546032 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:42.545981 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bk7xg"] Oct 30 12:50:42.546032 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:42.546028 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bk7xg" Oct 30 12:50:42.577849 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:42.577820 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 12:50:42.578035 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:42.577895 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 12:50:42.578087 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:42.578048 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-qj2wq\"" Oct 30 12:50:42.679160 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:42.679124 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7b709e7-8df1-4076-99c5-eef4415dd7fd-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bk7xg\" (UID: \"f7b709e7-8df1-4076-99c5-eef4415dd7fd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bk7xg" Oct 30 12:50:42.679363 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:42.679166 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfbv6\" (UniqueName: \"kubernetes.io/projected/f7b709e7-8df1-4076-99c5-eef4415dd7fd-kube-api-access-kfbv6\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bk7xg\" (UID: \"f7b709e7-8df1-4076-99c5-eef4415dd7fd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bk7xg" Oct 30 12:50:42.679453 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:42.679360 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7b709e7-8df1-4076-99c5-eef4415dd7fd-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bk7xg\" (UID: \"f7b709e7-8df1-4076-99c5-eef4415dd7fd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bk7xg" Oct 30 12:50:42.780641 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:42.780550 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7b709e7-8df1-4076-99c5-eef4415dd7fd-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bk7xg\" (UID: \"f7b709e7-8df1-4076-99c5-eef4415dd7fd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bk7xg" Oct 30 12:50:42.780641 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:42.780604 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7b709e7-8df1-4076-99c5-eef4415dd7fd-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bk7xg\" (UID: \"f7b709e7-8df1-4076-99c5-eef4415dd7fd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bk7xg" Oct 30 12:50:42.780641 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:42.780642 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfbv6\" (UniqueName: \"kubernetes.io/projected/f7b709e7-8df1-4076-99c5-eef4415dd7fd-kube-api-access-kfbv6\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bk7xg\" (UID: \"f7b709e7-8df1-4076-99c5-eef4415dd7fd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bk7xg" Oct 30 12:50:42.781027 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:42.781000 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7b709e7-8df1-4076-99c5-eef4415dd7fd-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bk7xg\" (UID: \"f7b709e7-8df1-4076-99c5-eef4415dd7fd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bk7xg" Oct 30 12:50:42.781027 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:42.781017 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7b709e7-8df1-4076-99c5-eef4415dd7fd-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bk7xg\" (UID: \"f7b709e7-8df1-4076-99c5-eef4415dd7fd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bk7xg" Oct 30 12:50:42.807335 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:42.807301 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfbv6\" (UniqueName: \"kubernetes.io/projected/f7b709e7-8df1-4076-99c5-eef4415dd7fd-kube-api-access-kfbv6\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bk7xg\" (UID: \"f7b709e7-8df1-4076-99c5-eef4415dd7fd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bk7xg" Oct 30 12:50:42.900903 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:42.900868 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bk7xg" Oct 30 12:50:43.231357 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:43.231324 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bk7xg"] Oct 30 12:50:43.234575 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:50:43.234540 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b709e7_8df1_4076_99c5_eef4415dd7fd.slice/crio-a94661b1d3c5c2c3149c676a17f658f84a9646e573668729fdce030a7f99612e WatchSource:0}: Error finding container a94661b1d3c5c2c3149c676a17f658f84a9646e573668729fdce030a7f99612e: Status 404 returned error can't find the container with id a94661b1d3c5c2c3149c676a17f658f84a9646e573668729fdce030a7f99612e Oct 30 12:50:43.884470 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:43.884432 4931 generic.go:356] "Generic (PLEG): container finished" podID="f7b709e7-8df1-4076-99c5-eef4415dd7fd" containerID="cb4bfca8b4a7c5d0bca3fef4eb3bd947cdac9d0e24ad9ee4f4b78300da0d43de" exitCode=0 Oct 30 12:50:43.884829 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:43.884522 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bk7xg" event={"ID":"f7b709e7-8df1-4076-99c5-eef4415dd7fd","Type":"ContainerDied","Data":"cb4bfca8b4a7c5d0bca3fef4eb3bd947cdac9d0e24ad9ee4f4b78300da0d43de"} Oct 30 12:50:43.884829 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:43.884578 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bk7xg" event={"ID":"f7b709e7-8df1-4076-99c5-eef4415dd7fd","Type":"ContainerStarted","Data":"a94661b1d3c5c2c3149c676a17f658f84a9646e573668729fdce030a7f99612e"} Oct 30 12:50:44.890557 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:44.890523 4931 generic.go:356] "Generic (PLEG): container finished" podID="f7b709e7-8df1-4076-99c5-eef4415dd7fd" containerID="69c31eef7c5ff482717ed929ade0e2e4f61829266ebefcdbe177a3741baee92b" exitCode=0 Oct 30 12:50:44.890962 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:44.890611 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bk7xg" event={"ID":"f7b709e7-8df1-4076-99c5-eef4415dd7fd","Type":"ContainerDied","Data":"69c31eef7c5ff482717ed929ade0e2e4f61829266ebefcdbe177a3741baee92b"} Oct 30 12:50:45.896786 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:45.896744 4931 generic.go:356] "Generic (PLEG): container finished" podID="f7b709e7-8df1-4076-99c5-eef4415dd7fd" containerID="3614fe4165e295e27ecb81310e4c0db43d713510f66321da165250cc1d1c2ea5" exitCode=0 Oct 30 12:50:45.897254 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:45.896794 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bk7xg" event={"ID":"f7b709e7-8df1-4076-99c5-eef4415dd7fd","Type":"ContainerDied","Data":"3614fe4165e295e27ecb81310e4c0db43d713510f66321da165250cc1d1c2ea5"} Oct 30 12:50:47.023463 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:47.023433 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bk7xg" Oct 30 12:50:47.118817 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:47.118779 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7b709e7-8df1-4076-99c5-eef4415dd7fd-kube-api-access-kfbv6\" (UniqueName: \"kubernetes.io/projected/f7b709e7-8df1-4076-99c5-eef4415dd7fd-kube-api-access-kfbv6\") pod \"f7b709e7-8df1-4076-99c5-eef4415dd7fd\" (UID: \"f7b709e7-8df1-4076-99c5-eef4415dd7fd\") " Oct 30 12:50:47.119007 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:47.118845 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b709e7-8df1-4076-99c5-eef4415dd7fd-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7b709e7-8df1-4076-99c5-eef4415dd7fd-bundle\") pod \"f7b709e7-8df1-4076-99c5-eef4415dd7fd\" (UID: \"f7b709e7-8df1-4076-99c5-eef4415dd7fd\") " Oct 30 12:50:47.119007 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:47.118883 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b709e7-8df1-4076-99c5-eef4415dd7fd-util\" (UniqueName: \"kubernetes.io/empty-dir/f7b709e7-8df1-4076-99c5-eef4415dd7fd-util\") pod \"f7b709e7-8df1-4076-99c5-eef4415dd7fd\" (UID: \"f7b709e7-8df1-4076-99c5-eef4415dd7fd\") " Oct 30 12:50:47.119469 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:47.119429 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b709e7-8df1-4076-99c5-eef4415dd7fd-bundle" pod "f7b709e7-8df1-4076-99c5-eef4415dd7fd" (UID: "f7b709e7-8df1-4076-99c5-eef4415dd7fd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:50:47.120899 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:47.120871 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b709e7-8df1-4076-99c5-eef4415dd7fd-kube-api-access-kfbv6" pod "f7b709e7-8df1-4076-99c5-eef4415dd7fd" (UID: "f7b709e7-8df1-4076-99c5-eef4415dd7fd"). InnerVolumeSpecName "kube-api-access-kfbv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:50:47.125050 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:47.125026 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b709e7-8df1-4076-99c5-eef4415dd7fd-util" pod "f7b709e7-8df1-4076-99c5-eef4415dd7fd" (UID: "f7b709e7-8df1-4076-99c5-eef4415dd7fd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:50:47.220087 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:47.220041 4931 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7b709e7-8df1-4076-99c5-eef4415dd7fd-util\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:50:47.220087 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:47.220086 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfbv6\" (UniqueName: \"kubernetes.io/projected/f7b709e7-8df1-4076-99c5-eef4415dd7fd-kube-api-access-kfbv6\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:50:47.220087 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:47.220099 4931 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7b709e7-8df1-4076-99c5-eef4415dd7fd-bundle\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:50:47.906895 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:47.906804 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bk7xg" event={"ID":"f7b709e7-8df1-4076-99c5-eef4415dd7fd","Type":"ContainerDied","Data":"a94661b1d3c5c2c3149c676a17f658f84a9646e573668729fdce030a7f99612e"} Oct 30 12:50:47.906895 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:47.906858 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a94661b1d3c5c2c3149c676a17f658f84a9646e573668729fdce030a7f99612e" Oct 30 12:50:47.906895 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:47.906825 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bk7xg" Oct 30 12:50:54.890513 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:54.890474 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c4494f88b-6wsdl"] Oct 30 12:50:54.890913 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:54.890768 4931 cpu_manager.go:414] "containerMap: removing container" podUID="f7b709e7-8df1-4076-99c5-eef4415dd7fd" containerName="pull" Oct 30 12:50:54.890913 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:54.890779 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b709e7-8df1-4076-99c5-eef4415dd7fd" containerName="pull" Oct 30 12:50:54.890913 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:54.890803 4931 cpu_manager.go:414] "containerMap: removing container" podUID="f7b709e7-8df1-4076-99c5-eef4415dd7fd" containerName="util" Oct 30 12:50:54.890913 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:54.890809 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b709e7-8df1-4076-99c5-eef4415dd7fd" containerName="util" Oct 30 12:50:54.890913 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:54.890815 4931 cpu_manager.go:414] "containerMap: removing container" podUID="f7b709e7-8df1-4076-99c5-eef4415dd7fd" containerName="extract" Oct 30 12:50:54.890913 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:54.890820 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b709e7-8df1-4076-99c5-eef4415dd7fd" containerName="extract" Oct 30 12:50:54.890913 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:54.890883 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7b709e7-8df1-4076-99c5-eef4415dd7fd" containerName="extract" Oct 30 12:50:54.893952 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:54.893934 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-6wsdl" Oct 30 12:50:54.916219 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:54.916195 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 12:50:54.916726 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:54.916708 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 12:50:54.928963 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:54.928941 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 12:50:54.929049 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:54.929022 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 12:50:54.929126 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:54.929109 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-fmghg\"" Oct 30 12:50:54.954763 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:54.954739 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c4494f88b-6wsdl"] Oct 30 12:50:55.016714 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:55.016693 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 12:50:55.084583 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:55.084550 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35aac1f4-f144-4455-97bb-6634f299bcd2-tmp\") pod \"opendatahub-operator-controller-manager-7c4494f88b-6wsdl\" (UID: \"35aac1f4-f144-4455-97bb-6634f299bcd2\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-6wsdl" Oct 30 12:50:55.084756 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:55.084587 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/35aac1f4-f144-4455-97bb-6634f299bcd2-metrics-cert\") pod \"opendatahub-operator-controller-manager-7c4494f88b-6wsdl\" (UID: \"35aac1f4-f144-4455-97bb-6634f299bcd2\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-6wsdl" Oct 30 12:50:55.084756 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:55.084606 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/35aac1f4-f144-4455-97bb-6634f299bcd2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c4494f88b-6wsdl\" (UID: \"35aac1f4-f144-4455-97bb-6634f299bcd2\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-6wsdl" Oct 30 12:50:55.084756 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:55.084621 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z7cn\" (UniqueName: \"kubernetes.io/projected/35aac1f4-f144-4455-97bb-6634f299bcd2-kube-api-access-8z7cn\") pod \"opendatahub-operator-controller-manager-7c4494f88b-6wsdl\" (UID: \"35aac1f4-f144-4455-97bb-6634f299bcd2\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-6wsdl" Oct 30 12:50:55.084904 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:55.084786 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35aac1f4-f144-4455-97bb-6634f299bcd2-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c4494f88b-6wsdl\" (UID: \"35aac1f4-f144-4455-97bb-6634f299bcd2\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-6wsdl" Oct 30 12:50:55.084904 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:55.084835 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/35aac1f4-f144-4455-97bb-6634f299bcd2-manifests\") pod \"opendatahub-operator-controller-manager-7c4494f88b-6wsdl\" (UID: \"35aac1f4-f144-4455-97bb-6634f299bcd2\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-6wsdl" Oct 30 12:50:55.185621 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:55.185533 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/35aac1f4-f144-4455-97bb-6634f299bcd2-metrics-cert\") pod \"opendatahub-operator-controller-manager-7c4494f88b-6wsdl\" (UID: \"35aac1f4-f144-4455-97bb-6634f299bcd2\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-6wsdl" Oct 30 12:50:55.185621 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:55.185571 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/35aac1f4-f144-4455-97bb-6634f299bcd2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c4494f88b-6wsdl\" (UID: \"35aac1f4-f144-4455-97bb-6634f299bcd2\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-6wsdl" Oct 30 12:50:55.185621 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:55.185587 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z7cn\" (UniqueName: \"kubernetes.io/projected/35aac1f4-f144-4455-97bb-6634f299bcd2-kube-api-access-8z7cn\") pod \"opendatahub-operator-controller-manager-7c4494f88b-6wsdl\" (UID: \"35aac1f4-f144-4455-97bb-6634f299bcd2\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-6wsdl" Oct 30 12:50:55.185621 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:55.185620 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35aac1f4-f144-4455-97bb-6634f299bcd2-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c4494f88b-6wsdl\" (UID: \"35aac1f4-f144-4455-97bb-6634f299bcd2\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-6wsdl" Oct 30 12:50:55.185938 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:55.185641 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/35aac1f4-f144-4455-97bb-6634f299bcd2-manifests\") pod \"opendatahub-operator-controller-manager-7c4494f88b-6wsdl\" (UID: \"35aac1f4-f144-4455-97bb-6634f299bcd2\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-6wsdl" Oct 30 12:50:55.185938 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:55.185755 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35aac1f4-f144-4455-97bb-6634f299bcd2-tmp\") pod \"opendatahub-operator-controller-manager-7c4494f88b-6wsdl\" (UID: \"35aac1f4-f144-4455-97bb-6634f299bcd2\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-6wsdl" Oct 30 12:50:55.186208 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:55.186178 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/35aac1f4-f144-4455-97bb-6634f299bcd2-manifests\") pod \"opendatahub-operator-controller-manager-7c4494f88b-6wsdl\" (UID: \"35aac1f4-f144-4455-97bb-6634f299bcd2\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-6wsdl" Oct 30 12:50:55.188070 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:55.188045 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35aac1f4-f144-4455-97bb-6634f299bcd2-tmp\") pod \"opendatahub-operator-controller-manager-7c4494f88b-6wsdl\" (UID: \"35aac1f4-f144-4455-97bb-6634f299bcd2\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-6wsdl" Oct 30 12:50:55.188153 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:55.188136 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35aac1f4-f144-4455-97bb-6634f299bcd2-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c4494f88b-6wsdl\" (UID: \"35aac1f4-f144-4455-97bb-6634f299bcd2\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-6wsdl" Oct 30 12:50:55.188192 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:55.188182 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/35aac1f4-f144-4455-97bb-6634f299bcd2-metrics-cert\") pod \"opendatahub-operator-controller-manager-7c4494f88b-6wsdl\" (UID: \"35aac1f4-f144-4455-97bb-6634f299bcd2\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-6wsdl" Oct 30 12:50:55.188231 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:55.188210 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/35aac1f4-f144-4455-97bb-6634f299bcd2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c4494f88b-6wsdl\" (UID: \"35aac1f4-f144-4455-97bb-6634f299bcd2\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-6wsdl" Oct 30 12:50:55.215896 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:55.215868 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z7cn\" (UniqueName: \"kubernetes.io/projected/35aac1f4-f144-4455-97bb-6634f299bcd2-kube-api-access-8z7cn\") pod \"opendatahub-operator-controller-manager-7c4494f88b-6wsdl\" (UID: \"35aac1f4-f144-4455-97bb-6634f299bcd2\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-6wsdl" Oct 30 12:50:55.508198 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:55.508162 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-6wsdl" Oct 30 12:50:55.876320 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:55.876295 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c4494f88b-6wsdl"] Oct 30 12:50:55.879096 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:50:55.879065 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35aac1f4_f144_4455_97bb_6634f299bcd2.slice/crio-446bad2cc0584331e59c3d81c8d92dae0a07ee9f6a69bef680fdf94f593f3ffd WatchSource:0}: Error finding container 446bad2cc0584331e59c3d81c8d92dae0a07ee9f6a69bef680fdf94f593f3ffd: Status 404 returned error can't find the container with id 446bad2cc0584331e59c3d81c8d92dae0a07ee9f6a69bef680fdf94f593f3ffd Oct 30 12:50:55.941903 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:55.941870 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-6wsdl" event={"ID":"35aac1f4-f144-4455-97bb-6634f299bcd2","Type":"ContainerStarted","Data":"446bad2cc0584331e59c3d81c8d92dae0a07ee9f6a69bef680fdf94f593f3ffd"} Oct 30 12:50:58.957106 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:58.957066 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-6wsdl" event={"ID":"35aac1f4-f144-4455-97bb-6634f299bcd2","Type":"ContainerStarted","Data":"fd36b308ac5d1de45d827108b4bad35991773330a173734b7d3b51527ea879d3"} Oct 30 12:50:59.962503 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:59.962468 4931 generic.go:356] "Generic (PLEG): container finished" podID="35aac1f4-f144-4455-97bb-6634f299bcd2" containerID="fd36b308ac5d1de45d827108b4bad35991773330a173734b7d3b51527ea879d3" exitCode=0 Oct 30 12:50:59.962901 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:50:59.962518 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-6wsdl" event={"ID":"35aac1f4-f144-4455-97bb-6634f299bcd2","Type":"ContainerDied","Data":"fd36b308ac5d1de45d827108b4bad35991773330a173734b7d3b51527ea879d3"} Oct 30 12:51:00.968638 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:00.968602 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-6wsdl" event={"ID":"35aac1f4-f144-4455-97bb-6634f299bcd2","Type":"ContainerStarted","Data":"7c6a6a9d7ee52b9289c4fbf337173ea220aa06b0a0cc92c14449970e41f2847e"} Oct 30 12:51:00.969026 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:00.968748 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-6wsdl" Oct 30 12:51:01.129867 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:01.129813 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-6wsdl" podStartSLOduration=3.646240452 podStartE2EDuration="7.129798935s" totalImagesPullingTime="3.023173446s" totalInitContainerRuntime="460.385037ms" isStatefulPod=false podCreationTimestamp="2026-10-30 12:50:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:01.115137837 +0000 UTC m=+359.718636283" watchObservedRunningTime="2026-10-30 12:51:01.129798935 +0000 UTC m=+359.733297381" Oct 30 12:51:07.351852 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:51:07.351810 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 12:51:11.975238 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:11.975204 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-6wsdl" Oct 30 12:51:12.393595 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:12.393508 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-55d66777d-h57kc"] Oct 30 12:51:12.400139 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:12.400117 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-55d66777d-h57kc" Oct 30 12:51:12.419797 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:12.419775 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6bbcdbb2-9991-404c-8ef8-2a09decd6500-metrics-cert\") pod \"lws-controller-manager-55d66777d-h57kc\" (UID: \"6bbcdbb2-9991-404c-8ef8-2a09decd6500\") " pod="openshift-lws-operator/lws-controller-manager-55d66777d-h57kc" Oct 30 12:51:12.419915 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:12.419813 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6bbcdbb2-9991-404c-8ef8-2a09decd6500-manager-config\") pod \"lws-controller-manager-55d66777d-h57kc\" (UID: \"6bbcdbb2-9991-404c-8ef8-2a09decd6500\") " pod="openshift-lws-operator/lws-controller-manager-55d66777d-h57kc" Oct 30 12:51:12.419955 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:12.419918 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6bbcdbb2-9991-404c-8ef8-2a09decd6500-cert\") pod \"lws-controller-manager-55d66777d-h57kc\" (UID: \"6bbcdbb2-9991-404c-8ef8-2a09decd6500\") " pod="openshift-lws-operator/lws-controller-manager-55d66777d-h57kc" Oct 30 12:51:12.419955 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:12.419935 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7vkx\" (UniqueName: \"kubernetes.io/projected/6bbcdbb2-9991-404c-8ef8-2a09decd6500-kube-api-access-j7vkx\") pod \"lws-controller-manager-55d66777d-h57kc\" (UID: \"6bbcdbb2-9991-404c-8ef8-2a09decd6500\") " pod="openshift-lws-operator/lws-controller-manager-55d66777d-h57kc" Oct 30 12:51:12.424277 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:12.424248 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Oct 30 12:51:12.424327 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:12.424289 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:51:12.425097 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:12.425082 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Oct 30 12:51:12.460559 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:12.460529 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-55d66777d-h57kc"] Oct 30 12:51:12.504605 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:12.504569 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Oct 30 12:51:12.504787 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:12.504611 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Oct 30 12:51:12.504787 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:12.504675 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-hw9dj\"" Oct 30 12:51:12.520279 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:12.520245 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6bbcdbb2-9991-404c-8ef8-2a09decd6500-cert\") pod \"lws-controller-manager-55d66777d-h57kc\" (UID: \"6bbcdbb2-9991-404c-8ef8-2a09decd6500\") " pod="openshift-lws-operator/lws-controller-manager-55d66777d-h57kc" Oct 30 12:51:12.520401 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:12.520305 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7vkx\" (UniqueName: \"kubernetes.io/projected/6bbcdbb2-9991-404c-8ef8-2a09decd6500-kube-api-access-j7vkx\") pod \"lws-controller-manager-55d66777d-h57kc\" (UID: \"6bbcdbb2-9991-404c-8ef8-2a09decd6500\") " pod="openshift-lws-operator/lws-controller-manager-55d66777d-h57kc" Oct 30 12:51:12.520401 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:12.520339 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6bbcdbb2-9991-404c-8ef8-2a09decd6500-metrics-cert\") pod \"lws-controller-manager-55d66777d-h57kc\" (UID: \"6bbcdbb2-9991-404c-8ef8-2a09decd6500\") " pod="openshift-lws-operator/lws-controller-manager-55d66777d-h57kc" Oct 30 12:51:12.520401 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:12.520365 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6bbcdbb2-9991-404c-8ef8-2a09decd6500-manager-config\") pod \"lws-controller-manager-55d66777d-h57kc\" (UID: \"6bbcdbb2-9991-404c-8ef8-2a09decd6500\") " pod="openshift-lws-operator/lws-controller-manager-55d66777d-h57kc" Oct 30 12:51:12.521020 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:12.520999 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6bbcdbb2-9991-404c-8ef8-2a09decd6500-manager-config\") pod \"lws-controller-manager-55d66777d-h57kc\" (UID: \"6bbcdbb2-9991-404c-8ef8-2a09decd6500\") " pod="openshift-lws-operator/lws-controller-manager-55d66777d-h57kc" Oct 30 12:51:12.522935 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:12.522911 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6bbcdbb2-9991-404c-8ef8-2a09decd6500-metrics-cert\") pod \"lws-controller-manager-55d66777d-h57kc\" (UID: \"6bbcdbb2-9991-404c-8ef8-2a09decd6500\") " pod="openshift-lws-operator/lws-controller-manager-55d66777d-h57kc" Oct 30 12:51:12.523012 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:12.522912 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6bbcdbb2-9991-404c-8ef8-2a09decd6500-cert\") pod \"lws-controller-manager-55d66777d-h57kc\" (UID: \"6bbcdbb2-9991-404c-8ef8-2a09decd6500\") " pod="openshift-lws-operator/lws-controller-manager-55d66777d-h57kc" Oct 30 12:51:12.547209 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:12.547172 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7vkx\" (UniqueName: \"kubernetes.io/projected/6bbcdbb2-9991-404c-8ef8-2a09decd6500-kube-api-access-j7vkx\") pod \"lws-controller-manager-55d66777d-h57kc\" (UID: \"6bbcdbb2-9991-404c-8ef8-2a09decd6500\") " pod="openshift-lws-operator/lws-controller-manager-55d66777d-h57kc" Oct 30 12:51:12.711112 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:12.711080 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-55d66777d-h57kc" Oct 30 12:51:13.072858 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:13.072803 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-55d66777d-h57kc"] Oct 30 12:51:13.075610 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:51:13.075584 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bbcdbb2_9991_404c_8ef8_2a09decd6500.slice/crio-873cec432d3905ce61ca0e98936a52ecdf16ac3366f4917ae6ec13f6293c37f6 WatchSource:0}: Error finding container 873cec432d3905ce61ca0e98936a52ecdf16ac3366f4917ae6ec13f6293c37f6: Status 404 returned error can't find the container with id 873cec432d3905ce61ca0e98936a52ecdf16ac3366f4917ae6ec13f6293c37f6 Oct 30 12:51:14.023684 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:14.023651 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-55d66777d-h57kc" event={"ID":"6bbcdbb2-9991-404c-8ef8-2a09decd6500","Type":"ContainerStarted","Data":"873cec432d3905ce61ca0e98936a52ecdf16ac3366f4917ae6ec13f6293c37f6"} Oct 30 12:51:15.029648 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:15.029608 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-55d66777d-h57kc" event={"ID":"6bbcdbb2-9991-404c-8ef8-2a09decd6500","Type":"ContainerStarted","Data":"f2c3f261e0e4db037b0594d92f9472e57af11dcec95e7cad65e67a1a6cf02161"} Oct 30 12:51:15.030090 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:15.029736 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-55d66777d-h57kc" Oct 30 12:51:15.094663 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:15.094608 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-55d66777d-h57kc" podStartSLOduration=1.668384014 podStartE2EDuration="3.094594523s" totalImagesPullingTime="1.426210509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:15.094456145 +0000 UTC m=+373.697954592" watchObservedRunningTime="2026-10-30 12:51:15.094594523 +0000 UTC m=+373.698092963" Oct 30 12:51:26.036355 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:26.036320 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-55d66777d-h57kc" Oct 30 12:51:31.851631 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:31.851588 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-psvs6"] Oct 30 12:51:31.858837 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:31.858811 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-psvs6" Oct 30 12:51:31.882451 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:31.882420 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-psvs6"] Oct 30 12:51:31.894644 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:31.894608 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-5xfqc\"" Oct 30 12:51:31.985536 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:31.985500 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 12:51:31.988732 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:31.988706 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c0cc6a6-f4f0-41c9-a6a4-522c19211b21-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-psvs6\" (UID: \"5c0cc6a6-f4f0-41c9-a6a4-522c19211b21\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-psvs6" Oct 30 12:51:31.988897 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:31.988747 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5c0cc6a6-f4f0-41c9-a6a4-522c19211b21-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-psvs6\" (UID: \"5c0cc6a6-f4f0-41c9-a6a4-522c19211b21\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-psvs6" Oct 30 12:51:31.988897 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:31.988774 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5c0cc6a6-f4f0-41c9-a6a4-522c19211b21-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-psvs6\" (UID: \"5c0cc6a6-f4f0-41c9-a6a4-522c19211b21\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-psvs6" Oct 30 12:51:31.988897 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:31.988887 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxdbx\" (UniqueName: \"kubernetes.io/projected/5c0cc6a6-f4f0-41c9-a6a4-522c19211b21-kube-api-access-bxdbx\") pod \"kserve-module-controller-manager-5cfb77bf8c-psvs6\" (UID: \"5c0cc6a6-f4f0-41c9-a6a4-522c19211b21\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-psvs6" Oct 30 12:51:32.090173 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:32.090135 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5c0cc6a6-f4f0-41c9-a6a4-522c19211b21-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-psvs6\" (UID: \"5c0cc6a6-f4f0-41c9-a6a4-522c19211b21\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-psvs6" Oct 30 12:51:32.090350 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:32.090179 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxdbx\" (UniqueName: \"kubernetes.io/projected/5c0cc6a6-f4f0-41c9-a6a4-522c19211b21-kube-api-access-bxdbx\") pod \"kserve-module-controller-manager-5cfb77bf8c-psvs6\" (UID: \"5c0cc6a6-f4f0-41c9-a6a4-522c19211b21\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-psvs6" Oct 30 12:51:32.090350 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:32.090306 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c0cc6a6-f4f0-41c9-a6a4-522c19211b21-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-psvs6\" (UID: \"5c0cc6a6-f4f0-41c9-a6a4-522c19211b21\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-psvs6" Oct 30 12:51:32.090430 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:32.090356 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5c0cc6a6-f4f0-41c9-a6a4-522c19211b21-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-psvs6\" (UID: \"5c0cc6a6-f4f0-41c9-a6a4-522c19211b21\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-psvs6" Oct 30 12:51:32.090736 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:32.090716 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c0cc6a6-f4f0-41c9-a6a4-522c19211b21-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-psvs6\" (UID: \"5c0cc6a6-f4f0-41c9-a6a4-522c19211b21\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-psvs6" Oct 30 12:51:32.090786 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:32.090766 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5c0cc6a6-f4f0-41c9-a6a4-522c19211b21-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-psvs6\" (UID: \"5c0cc6a6-f4f0-41c9-a6a4-522c19211b21\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-psvs6" Oct 30 12:51:32.091123 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:32.091104 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5c0cc6a6-f4f0-41c9-a6a4-522c19211b21-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-psvs6\" (UID: \"5c0cc6a6-f4f0-41c9-a6a4-522c19211b21\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-psvs6" Oct 30 12:51:32.115372 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:32.115306 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxdbx\" (UniqueName: \"kubernetes.io/projected/5c0cc6a6-f4f0-41c9-a6a4-522c19211b21-kube-api-access-bxdbx\") pod \"kserve-module-controller-manager-5cfb77bf8c-psvs6\" (UID: \"5c0cc6a6-f4f0-41c9-a6a4-522c19211b21\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-psvs6" Oct 30 12:51:32.175068 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:32.175035 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-psvs6" Oct 30 12:51:32.502338 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:32.502300 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-psvs6"] Oct 30 12:51:32.506354 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:51:32.506325 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c0cc6a6_f4f0_41c9_a6a4_522c19211b21.slice/crio-51320661df592e03e1ae3afe80fce166d6142a6b2fdc29beb43497750130524e WatchSource:0}: Error finding container 51320661df592e03e1ae3afe80fce166d6142a6b2fdc29beb43497750130524e: Status 404 returned error can't find the container with id 51320661df592e03e1ae3afe80fce166d6142a6b2fdc29beb43497750130524e Oct 30 12:51:33.110478 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:33.110428 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-psvs6" event={"ID":"5c0cc6a6-f4f0-41c9-a6a4-522c19211b21","Type":"ContainerStarted","Data":"51320661df592e03e1ae3afe80fce166d6142a6b2fdc29beb43497750130524e"} Oct 30 12:51:34.032240 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:34.030875 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8"] Oct 30 12:51:34.039832 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:34.039805 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" Oct 30 12:51:34.059174 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:34.059145 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8"] Oct 30 12:51:34.059375 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:34.059358 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 12:51:34.071146 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:34.071119 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-6ckp5\"" Oct 30 12:51:34.105936 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:34.105894 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4a08295-1052-4b1c-86ff-ca8729ed4dd8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8\" (UID: \"e4a08295-1052-4b1c-86ff-ca8729ed4dd8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" Oct 30 12:51:34.106113 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:34.105947 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4a08295-1052-4b1c-86ff-ca8729ed4dd8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8\" (UID: \"e4a08295-1052-4b1c-86ff-ca8729ed4dd8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" Oct 30 12:51:34.106113 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:34.105982 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4a08295-1052-4b1c-86ff-ca8729ed4dd8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8\" (UID: \"e4a08295-1052-4b1c-86ff-ca8729ed4dd8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" Oct 30 12:51:34.106113 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:34.106103 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4a08295-1052-4b1c-86ff-ca8729ed4dd8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8\" (UID: \"e4a08295-1052-4b1c-86ff-ca8729ed4dd8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" Oct 30 12:51:34.106308 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:34.106156 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4a08295-1052-4b1c-86ff-ca8729ed4dd8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8\" (UID: \"e4a08295-1052-4b1c-86ff-ca8729ed4dd8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" Oct 30 12:51:34.106308 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:34.106187 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4a08295-1052-4b1c-86ff-ca8729ed4dd8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8\" (UID: \"e4a08295-1052-4b1c-86ff-ca8729ed4dd8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" Oct 30 12:51:34.106308 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:34.106245 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4a08295-1052-4b1c-86ff-ca8729ed4dd8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8\" (UID: \"e4a08295-1052-4b1c-86ff-ca8729ed4dd8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" Oct 30 12:51:34.106466 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:34.106317 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4a08295-1052-4b1c-86ff-ca8729ed4dd8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8\" (UID: \"e4a08295-1052-4b1c-86ff-ca8729ed4dd8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" Oct 30 12:51:34.106466 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:34.106349 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg9r4\" (UniqueName: \"kubernetes.io/projected/e4a08295-1052-4b1c-86ff-ca8729ed4dd8-kube-api-access-kg9r4\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8\" (UID: \"e4a08295-1052-4b1c-86ff-ca8729ed4dd8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" Oct 30 12:51:34.207695 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:34.207652 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4a08295-1052-4b1c-86ff-ca8729ed4dd8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8\" (UID: \"e4a08295-1052-4b1c-86ff-ca8729ed4dd8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" Oct 30 12:51:34.208210 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:34.207702 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4a08295-1052-4b1c-86ff-ca8729ed4dd8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8\" (UID: \"e4a08295-1052-4b1c-86ff-ca8729ed4dd8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" Oct 30 12:51:34.208210 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:34.207760 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4a08295-1052-4b1c-86ff-ca8729ed4dd8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8\" (UID: \"e4a08295-1052-4b1c-86ff-ca8729ed4dd8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" Oct 30 12:51:34.208210 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:34.207816 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4a08295-1052-4b1c-86ff-ca8729ed4dd8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8\" (UID: \"e4a08295-1052-4b1c-86ff-ca8729ed4dd8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" Oct 30 12:51:34.208210 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:34.207845 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg9r4\" (UniqueName: \"kubernetes.io/projected/e4a08295-1052-4b1c-86ff-ca8729ed4dd8-kube-api-access-kg9r4\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8\" (UID: \"e4a08295-1052-4b1c-86ff-ca8729ed4dd8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" Oct 30 12:51:34.208210 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:34.207876 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4a08295-1052-4b1c-86ff-ca8729ed4dd8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8\" (UID: \"e4a08295-1052-4b1c-86ff-ca8729ed4dd8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" Oct 30 12:51:34.208210 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:34.207900 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4a08295-1052-4b1c-86ff-ca8729ed4dd8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8\" (UID: \"e4a08295-1052-4b1c-86ff-ca8729ed4dd8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" Oct 30 12:51:34.208210 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:34.207941 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4a08295-1052-4b1c-86ff-ca8729ed4dd8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8\" (UID: \"e4a08295-1052-4b1c-86ff-ca8729ed4dd8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" Oct 30 12:51:34.208210 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:34.207978 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4a08295-1052-4b1c-86ff-ca8729ed4dd8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8\" (UID: \"e4a08295-1052-4b1c-86ff-ca8729ed4dd8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" Oct 30 12:51:34.208210 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:34.208110 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4a08295-1052-4b1c-86ff-ca8729ed4dd8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8\" (UID: \"e4a08295-1052-4b1c-86ff-ca8729ed4dd8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" Oct 30 12:51:34.209144 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:34.208638 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4a08295-1052-4b1c-86ff-ca8729ed4dd8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8\" (UID: \"e4a08295-1052-4b1c-86ff-ca8729ed4dd8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" Oct 30 12:51:34.209144 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:34.208689 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4a08295-1052-4b1c-86ff-ca8729ed4dd8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8\" (UID: \"e4a08295-1052-4b1c-86ff-ca8729ed4dd8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" Oct 30 12:51:34.209144 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:34.208744 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4a08295-1052-4b1c-86ff-ca8729ed4dd8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8\" (UID: \"e4a08295-1052-4b1c-86ff-ca8729ed4dd8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" Oct 30 12:51:34.209144 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:34.208917 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4a08295-1052-4b1c-86ff-ca8729ed4dd8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8\" (UID: \"e4a08295-1052-4b1c-86ff-ca8729ed4dd8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" Oct 30 12:51:34.210501 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:34.210465 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4a08295-1052-4b1c-86ff-ca8729ed4dd8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8\" (UID: \"e4a08295-1052-4b1c-86ff-ca8729ed4dd8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" Oct 30 12:51:34.210864 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:34.210840 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4a08295-1052-4b1c-86ff-ca8729ed4dd8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8\" (UID: \"e4a08295-1052-4b1c-86ff-ca8729ed4dd8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" Oct 30 12:51:34.234049 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:34.234019 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4a08295-1052-4b1c-86ff-ca8729ed4dd8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8\" (UID: \"e4a08295-1052-4b1c-86ff-ca8729ed4dd8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" Oct 30 12:51:34.234158 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:34.234084 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg9r4\" (UniqueName: \"kubernetes.io/projected/e4a08295-1052-4b1c-86ff-ca8729ed4dd8-kube-api-access-kg9r4\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8\" (UID: \"e4a08295-1052-4b1c-86ff-ca8729ed4dd8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" Oct 30 12:51:34.355446 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:34.355363 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" Oct 30 12:51:34.684472 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:34.684447 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8"] Oct 30 12:51:34.686353 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:51:34.686324 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4a08295_1052_4b1c_86ff_ca8729ed4dd8.slice/crio-1b441c15f5357cae3aa22ff62568bae6080ac0df4f855961f628f7dc7c79c55c WatchSource:0}: Error finding container 1b441c15f5357cae3aa22ff62568bae6080ac0df4f855961f628f7dc7c79c55c: Status 404 returned error can't find the container with id 1b441c15f5357cae3aa22ff62568bae6080ac0df4f855961f628f7dc7c79c55c Oct 30 12:51:35.120869 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:35.120829 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-psvs6" event={"ID":"5c0cc6a6-f4f0-41c9-a6a4-522c19211b21","Type":"ContainerStarted","Data":"cf2c30bcd12e6d909f42863e0d194c21b9b10ce0b28032961a0638a0902cb6c1"} Oct 30 12:51:35.121068 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:35.120918 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-psvs6" Oct 30 12:51:35.122103 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:35.122075 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" event={"ID":"e4a08295-1052-4b1c-86ff-ca8729ed4dd8","Type":"ContainerStarted","Data":"1b441c15f5357cae3aa22ff62568bae6080ac0df4f855961f628f7dc7c79c55c"} Oct 30 12:51:35.168426 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:35.168368 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-psvs6" podStartSLOduration=1.951903239 podStartE2EDuration="4.168355239s" totalImagesPullingTime="2.216452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:35.16756677 +0000 UTC m=+393.771065215" watchObservedRunningTime="2026-10-30 12:51:35.168355239 +0000 UTC m=+393.771853685" Oct 30 12:51:37.249853 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:37.249817 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:51:37.250164 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:37.249889 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:51:37.250164 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:37.249919 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:51:37.354171 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:51:37.354134 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 12:51:38.137467 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:38.137432 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" event={"ID":"e4a08295-1052-4b1c-86ff-ca8729ed4dd8","Type":"ContainerStarted","Data":"f2d9b45ad21e194254660e7ec983cee8d03228ef563e31d9cf3420a49031b779"} Oct 30 12:51:38.212994 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:38.212939 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" podStartSLOduration=1.652302927 podStartE2EDuration="4.212924429s" totalImagesPullingTime="2.560621502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:38.211216551 +0000 UTC m=+396.814714994" watchObservedRunningTime="2026-10-30 12:51:38.212924429 +0000 UTC m=+396.816422874" Oct 30 12:51:38.356313 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:38.356250 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" Oct 30 12:51:38.361317 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:38.361290 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" Oct 30 12:51:39.141525 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:39.141495 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" Oct 30 12:51:39.142376 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:39.142362 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8" Oct 30 12:51:56.129362 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:51:56.129323 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-psvs6" Oct 30 12:52:07.353004 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:52:07.352959 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 12:52:20.717318 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:20.717281 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-l879q"] Oct 30 12:52:20.724063 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:20.724042 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-l879q" Oct 30 12:52:20.742906 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:20.742881 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-l879q"] Oct 30 12:52:20.756765 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:20.756738 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-c6n2t"] Oct 30 12:52:20.760229 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:20.760210 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-c6n2t" Oct 30 12:52:20.774712 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:20.774690 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2dc4e69c-b092-4f23-99f5-64b9436a5a6c-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-c6n2t\" (UID: \"2dc4e69c-b092-4f23-99f5-64b9436a5a6c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-c6n2t" Oct 30 12:52:20.774830 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:20.774734 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zcgl\" (UniqueName: \"kubernetes.io/projected/2dc4e69c-b092-4f23-99f5-64b9436a5a6c-kube-api-access-2zcgl\") pod \"llmisvc-controller-manager-869d5dcc5d-c6n2t\" (UID: \"2dc4e69c-b092-4f23-99f5-64b9436a5a6c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-c6n2t" Oct 30 12:52:20.800602 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:20.800575 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-c6n2t"] Oct 30 12:52:20.817890 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:20.817864 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-scw4n"] Oct 30 12:52:20.821559 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:20.821540 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-scw4n" Oct 30 12:52:20.844405 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:20.844379 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-zddwp\"" Oct 30 12:52:20.844538 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:20.844442 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-hpqpg\"" Oct 30 12:52:20.844951 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:20.844930 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 12:52:20.844951 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:20.844938 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 12:52:20.846655 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:20.846623 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-scw4n"] Oct 30 12:52:20.875241 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:20.875215 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zcgl\" (UniqueName: \"kubernetes.io/projected/2dc4e69c-b092-4f23-99f5-64b9436a5a6c-kube-api-access-2zcgl\") pod \"llmisvc-controller-manager-869d5dcc5d-c6n2t\" (UID: \"2dc4e69c-b092-4f23-99f5-64b9436a5a6c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-c6n2t" Oct 30 12:52:20.875389 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:20.875248 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjfnf\" (UniqueName: \"kubernetes.io/projected/75424157-54c3-41fc-8e08-1f2a976e254c-kube-api-access-zjfnf\") pod \"kserve-controller-manager-776cc66d47-l879q\" (UID: \"75424157-54c3-41fc-8e08-1f2a976e254c\") " pod="opendatahub/kserve-controller-manager-776cc66d47-l879q" Oct 30 12:52:20.875389 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:20.875286 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8n2l\" (UniqueName: \"kubernetes.io/projected/888aade1-5070-4097-8127-43a53ce4a3c3-kube-api-access-g8n2l\") pod \"model-serving-api-75bd89fdfc-scw4n\" (UID: \"888aade1-5070-4097-8127-43a53ce4a3c3\") " pod="opendatahub/model-serving-api-75bd89fdfc-scw4n" Oct 30 12:52:20.875463 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:20.875412 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/888aade1-5070-4097-8127-43a53ce4a3c3-tls-certs\") pod \"model-serving-api-75bd89fdfc-scw4n\" (UID: \"888aade1-5070-4097-8127-43a53ce4a3c3\") " pod="opendatahub/model-serving-api-75bd89fdfc-scw4n" Oct 30 12:52:20.875514 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:20.875495 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/75424157-54c3-41fc-8e08-1f2a976e254c-cert\") pod \"kserve-controller-manager-776cc66d47-l879q\" (UID: \"75424157-54c3-41fc-8e08-1f2a976e254c\") " pod="opendatahub/kserve-controller-manager-776cc66d47-l879q" Oct 30 12:52:20.875555 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:20.875540 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2dc4e69c-b092-4f23-99f5-64b9436a5a6c-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-c6n2t\" (UID: \"2dc4e69c-b092-4f23-99f5-64b9436a5a6c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-c6n2t" Oct 30 12:52:20.876863 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:20.876823 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-jq9w7\"" Oct 30 12:52:20.877119 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:20.877103 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 12:52:20.878072 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:20.878052 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2dc4e69c-b092-4f23-99f5-64b9436a5a6c-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-c6n2t\" (UID: \"2dc4e69c-b092-4f23-99f5-64b9436a5a6c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-c6n2t" Oct 30 12:52:20.908025 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:20.907999 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zcgl\" (UniqueName: \"kubernetes.io/projected/2dc4e69c-b092-4f23-99f5-64b9436a5a6c-kube-api-access-2zcgl\") pod \"llmisvc-controller-manager-869d5dcc5d-c6n2t\" (UID: \"2dc4e69c-b092-4f23-99f5-64b9436a5a6c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-c6n2t" Oct 30 12:52:20.976657 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:20.976579 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/888aade1-5070-4097-8127-43a53ce4a3c3-tls-certs\") pod \"model-serving-api-75bd89fdfc-scw4n\" (UID: \"888aade1-5070-4097-8127-43a53ce4a3c3\") " pod="opendatahub/model-serving-api-75bd89fdfc-scw4n" Oct 30 12:52:20.976657 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:20.976622 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/75424157-54c3-41fc-8e08-1f2a976e254c-cert\") pod \"kserve-controller-manager-776cc66d47-l879q\" (UID: \"75424157-54c3-41fc-8e08-1f2a976e254c\") " pod="opendatahub/kserve-controller-manager-776cc66d47-l879q" Oct 30 12:52:20.976873 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:20.976662 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjfnf\" (UniqueName: \"kubernetes.io/projected/75424157-54c3-41fc-8e08-1f2a976e254c-kube-api-access-zjfnf\") pod \"kserve-controller-manager-776cc66d47-l879q\" (UID: \"75424157-54c3-41fc-8e08-1f2a976e254c\") " pod="opendatahub/kserve-controller-manager-776cc66d47-l879q" Oct 30 12:52:20.976873 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:20.976684 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8n2l\" (UniqueName: \"kubernetes.io/projected/888aade1-5070-4097-8127-43a53ce4a3c3-kube-api-access-g8n2l\") pod \"model-serving-api-75bd89fdfc-scw4n\" (UID: \"888aade1-5070-4097-8127-43a53ce4a3c3\") " pod="opendatahub/model-serving-api-75bd89fdfc-scw4n" Oct 30 12:52:20.979023 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:20.978997 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/75424157-54c3-41fc-8e08-1f2a976e254c-cert\") pod \"kserve-controller-manager-776cc66d47-l879q\" (UID: \"75424157-54c3-41fc-8e08-1f2a976e254c\") " pod="opendatahub/kserve-controller-manager-776cc66d47-l879q" Oct 30 12:52:20.979023 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:20.979013 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/888aade1-5070-4097-8127-43a53ce4a3c3-tls-certs\") pod \"model-serving-api-75bd89fdfc-scw4n\" (UID: \"888aade1-5070-4097-8127-43a53ce4a3c3\") " pod="opendatahub/model-serving-api-75bd89fdfc-scw4n" Oct 30 12:52:21.005441 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:21.005416 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjfnf\" (UniqueName: \"kubernetes.io/projected/75424157-54c3-41fc-8e08-1f2a976e254c-kube-api-access-zjfnf\") pod \"kserve-controller-manager-776cc66d47-l879q\" (UID: \"75424157-54c3-41fc-8e08-1f2a976e254c\") " pod="opendatahub/kserve-controller-manager-776cc66d47-l879q" Oct 30 12:52:21.009970 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:21.009948 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8n2l\" (UniqueName: \"kubernetes.io/projected/888aade1-5070-4097-8127-43a53ce4a3c3-kube-api-access-g8n2l\") pod \"model-serving-api-75bd89fdfc-scw4n\" (UID: \"888aade1-5070-4097-8127-43a53ce4a3c3\") " pod="opendatahub/model-serving-api-75bd89fdfc-scw4n" Oct 30 12:52:21.035753 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:21.035733 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-l879q" Oct 30 12:52:21.074712 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:21.074669 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-c6n2t" Oct 30 12:52:21.135078 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:21.134474 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-scw4n" Oct 30 12:52:21.333798 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:21.333775 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-l879q"] Oct 30 12:52:21.336249 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:52:21.336224 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75424157_54c3_41fc_8e08_1f2a976e254c.slice/crio-bdbb3abf4b7eace42f8a1290f784112ed484ef893800cc5f814ac17e4f4dd553 WatchSource:0}: Error finding container bdbb3abf4b7eace42f8a1290f784112ed484ef893800cc5f814ac17e4f4dd553: Status 404 returned error can't find the container with id bdbb3abf4b7eace42f8a1290f784112ed484ef893800cc5f814ac17e4f4dd553 Oct 30 12:52:21.382871 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:21.382846 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-c6n2t"] Oct 30 12:52:21.384817 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:52:21.384792 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2dc4e69c_b092_4f23_99f5_64b9436a5a6c.slice/crio-c9804bb4ee3128714dd084dbd1e64e9e07a7d74da16cec492ccdf8cc53c5bf4f WatchSource:0}: Error finding container c9804bb4ee3128714dd084dbd1e64e9e07a7d74da16cec492ccdf8cc53c5bf4f: Status 404 returned error can't find the container with id c9804bb4ee3128714dd084dbd1e64e9e07a7d74da16cec492ccdf8cc53c5bf4f Oct 30 12:52:21.435057 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:21.435030 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-scw4n"] Oct 30 12:52:21.437535 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:52:21.437509 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod888aade1_5070_4097_8127_43a53ce4a3c3.slice/crio-a6398be7255574ae0caef3ea9b2cdf46e86f96affe81e5dd5a655b208406a9b7 WatchSource:0}: Error finding container a6398be7255574ae0caef3ea9b2cdf46e86f96affe81e5dd5a655b208406a9b7: Status 404 returned error can't find the container with id a6398be7255574ae0caef3ea9b2cdf46e86f96affe81e5dd5a655b208406a9b7 Oct 30 12:52:22.338093 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:22.338031 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-scw4n" event={"ID":"888aade1-5070-4097-8127-43a53ce4a3c3","Type":"ContainerStarted","Data":"a6398be7255574ae0caef3ea9b2cdf46e86f96affe81e5dd5a655b208406a9b7"} Oct 30 12:52:22.339640 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:22.339611 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-l879q" event={"ID":"75424157-54c3-41fc-8e08-1f2a976e254c","Type":"ContainerStarted","Data":"bdbb3abf4b7eace42f8a1290f784112ed484ef893800cc5f814ac17e4f4dd553"} Oct 30 12:52:22.341537 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:22.341488 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-c6n2t" event={"ID":"2dc4e69c-b092-4f23-99f5-64b9436a5a6c","Type":"ContainerStarted","Data":"c9804bb4ee3128714dd084dbd1e64e9e07a7d74da16cec492ccdf8cc53c5bf4f"} Oct 30 12:52:24.355208 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:24.355167 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-l879q" event={"ID":"75424157-54c3-41fc-8e08-1f2a976e254c","Type":"ContainerStarted","Data":"5247a0c230367e8925f2be827627292e641f5a38ba77045fe6fa5710e6a53f7a"} Oct 30 12:52:24.355721 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:24.355298 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-l879q" Oct 30 12:52:24.356889 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:24.356858 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-c6n2t" event={"ID":"2dc4e69c-b092-4f23-99f5-64b9436a5a6c","Type":"ContainerStarted","Data":"ec9796fc60a0c524c41eb746df0724902e7d19c6a4a3128a40e2950cc3937beb"} Oct 30 12:52:24.357038 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:24.356914 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-c6n2t" Oct 30 12:52:24.358307 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:24.358286 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-scw4n" event={"ID":"888aade1-5070-4097-8127-43a53ce4a3c3","Type":"ContainerStarted","Data":"1731ae4f78ad5f295165181762069b8b578b2caf5ffd1ab452e10d3946e5fc3c"} Oct 30 12:52:24.358437 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:24.358424 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-scw4n" Oct 30 12:52:24.400626 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:24.400580 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-l879q" podStartSLOduration=1.931313486 podStartE2EDuration="4.400568492s" totalImagesPullingTime="2.469255006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:24.400029046 +0000 UTC m=+443.003527492" watchObservedRunningTime="2026-10-30 12:52:24.400568492 +0000 UTC m=+443.004066938" Oct 30 12:52:24.447325 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:24.447252 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-scw4n" podStartSLOduration=2.547957194 podStartE2EDuration="4.44723902s" totalImagesPullingTime="1.899281826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:24.446205749 +0000 UTC m=+443.049704194" watchObservedRunningTime="2026-10-30 12:52:24.44723902 +0000 UTC m=+443.050737465" Oct 30 12:52:24.494676 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:24.494621 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-c6n2t" podStartSLOduration=2.256011293 podStartE2EDuration="4.494607287s" totalImagesPullingTime="2.238595994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:24.492722472 +0000 UTC m=+443.096220918" watchObservedRunningTime="2026-10-30 12:52:24.494607287 +0000 UTC m=+443.098105733" Oct 30 12:52:35.366747 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:35.366713 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-scw4n" Oct 30 12:52:37.356399 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:52:37.356359 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 12:52:55.365291 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:55.365235 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-c6n2t" Oct 30 12:52:55.369490 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:52:55.369461 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-l879q" Oct 30 12:53:07.350514 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:53:07.350472 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 12:53:11.461078 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.461042 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z97zc"] Oct 30 12:53:11.465426 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.465405 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z97zc" Oct 30 12:53:11.497992 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.497963 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 12:53:11.498211 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.498011 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 12:53:11.498211 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.498025 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-cfmpg\"" Oct 30 12:53:11.499793 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.499773 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z97zc"] Oct 30 12:53:11.558913 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.558883 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7jgp"] Oct 30 12:53:11.562115 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.562097 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7jgp" Oct 30 12:53:11.583238 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.583210 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7jgp"] Oct 30 12:53:11.584061 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.584031 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-8ms8t\"" Oct 30 12:53:11.593424 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.593397 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qvlf\" (UniqueName: \"kubernetes.io/projected/151643ed-3090-4391-a8ab-53243381fb2a-kube-api-access-6qvlf\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z97zc\" (UID: \"151643ed-3090-4391-a8ab-53243381fb2a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z97zc" Oct 30 12:53:11.593572 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.593440 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/151643ed-3090-4391-a8ab-53243381fb2a-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z97zc\" (UID: \"151643ed-3090-4391-a8ab-53243381fb2a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z97zc" Oct 30 12:53:11.593572 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.593534 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/151643ed-3090-4391-a8ab-53243381fb2a-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z97zc\" (UID: \"151643ed-3090-4391-a8ab-53243381fb2a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z97zc" Oct 30 12:53:11.653457 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.653427 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs7dt"] Oct 30 12:53:11.656697 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.656679 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs7dt" Oct 30 12:53:11.675685 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.675657 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-n2v5c\"" Oct 30 12:53:11.677531 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.677510 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs7dt"] Oct 30 12:53:11.694622 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.694600 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/151643ed-3090-4391-a8ab-53243381fb2a-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z97zc\" (UID: \"151643ed-3090-4391-a8ab-53243381fb2a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z97zc" Oct 30 12:53:11.694722 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.694637 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktsvv\" (UniqueName: \"kubernetes.io/projected/4ba1b363-bee4-4589-b486-135b70fa04a4-kube-api-access-ktsvv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7jgp\" (UID: \"4ba1b363-bee4-4589-b486-135b70fa04a4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7jgp" Oct 30 12:53:11.694775 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.694734 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ba1b363-bee4-4589-b486-135b70fa04a4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7jgp\" (UID: \"4ba1b363-bee4-4589-b486-135b70fa04a4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7jgp" Oct 30 12:53:11.694775 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.694766 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qvlf\" (UniqueName: \"kubernetes.io/projected/151643ed-3090-4391-a8ab-53243381fb2a-kube-api-access-6qvlf\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z97zc\" (UID: \"151643ed-3090-4391-a8ab-53243381fb2a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z97zc" Oct 30 12:53:11.694884 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.694825 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ba1b363-bee4-4589-b486-135b70fa04a4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7jgp\" (UID: \"4ba1b363-bee4-4589-b486-135b70fa04a4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7jgp" Oct 30 12:53:11.694936 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.694914 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/151643ed-3090-4391-a8ab-53243381fb2a-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z97zc\" (UID: \"151643ed-3090-4391-a8ab-53243381fb2a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z97zc" Oct 30 12:53:11.695038 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.695004 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/151643ed-3090-4391-a8ab-53243381fb2a-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z97zc\" (UID: \"151643ed-3090-4391-a8ab-53243381fb2a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z97zc" Oct 30 12:53:11.695179 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.695164 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/151643ed-3090-4391-a8ab-53243381fb2a-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z97zc\" (UID: \"151643ed-3090-4391-a8ab-53243381fb2a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z97zc" Oct 30 12:53:11.721457 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.721393 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qvlf\" (UniqueName: \"kubernetes.io/projected/151643ed-3090-4391-a8ab-53243381fb2a-kube-api-access-6qvlf\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z97zc\" (UID: \"151643ed-3090-4391-a8ab-53243381fb2a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z97zc" Oct 30 12:53:11.750364 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.750333 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbrmqw"] Oct 30 12:53:11.753818 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.753801 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbrmqw" Oct 30 12:53:11.770180 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.770154 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-hh7dm\"" Oct 30 12:53:11.775240 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.775217 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbrmqw"] Oct 30 12:53:11.779631 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.779613 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z97zc" Oct 30 12:53:11.796406 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.796379 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ba1b363-bee4-4589-b486-135b70fa04a4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7jgp\" (UID: \"4ba1b363-bee4-4589-b486-135b70fa04a4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7jgp" Oct 30 12:53:11.796504 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.796426 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ba1b363-bee4-4589-b486-135b70fa04a4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7jgp\" (UID: \"4ba1b363-bee4-4589-b486-135b70fa04a4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7jgp" Oct 30 12:53:11.796504 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.796471 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e89365a9-1412-4108-b989-4fed047ed479-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs7dt\" (UID: \"e89365a9-1412-4108-b989-4fed047ed479\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs7dt" Oct 30 12:53:11.796628 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.796507 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e89365a9-1412-4108-b989-4fed047ed479-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs7dt\" (UID: \"e89365a9-1412-4108-b989-4fed047ed479\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs7dt" Oct 30 12:53:11.796628 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.796621 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktsvv\" (UniqueName: \"kubernetes.io/projected/4ba1b363-bee4-4589-b486-135b70fa04a4-kube-api-access-ktsvv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7jgp\" (UID: \"4ba1b363-bee4-4589-b486-135b70fa04a4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7jgp" Oct 30 12:53:11.796744 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.796665 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s25qz\" (UniqueName: \"kubernetes.io/projected/e89365a9-1412-4108-b989-4fed047ed479-kube-api-access-s25qz\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs7dt\" (UID: \"e89365a9-1412-4108-b989-4fed047ed479\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs7dt" Oct 30 12:53:11.796830 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.796811 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ba1b363-bee4-4589-b486-135b70fa04a4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7jgp\" (UID: \"4ba1b363-bee4-4589-b486-135b70fa04a4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7jgp" Oct 30 12:53:11.796884 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.796845 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ba1b363-bee4-4589-b486-135b70fa04a4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7jgp\" (UID: \"4ba1b363-bee4-4589-b486-135b70fa04a4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7jgp" Oct 30 12:53:11.825248 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.825216 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktsvv\" (UniqueName: \"kubernetes.io/projected/4ba1b363-bee4-4589-b486-135b70fa04a4-kube-api-access-ktsvv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7jgp\" (UID: \"4ba1b363-bee4-4589-b486-135b70fa04a4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7jgp" Oct 30 12:53:11.876978 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.876940 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7jgp" Oct 30 12:53:11.897080 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.897055 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e89365a9-1412-4108-b989-4fed047ed479-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs7dt\" (UID: \"e89365a9-1412-4108-b989-4fed047ed479\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs7dt" Oct 30 12:53:11.897194 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.897091 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbrmqw\" (UID: \"fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbrmqw" Oct 30 12:53:11.897253 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.897229 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s25qz\" (UniqueName: \"kubernetes.io/projected/e89365a9-1412-4108-b989-4fed047ed479-kube-api-access-s25qz\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs7dt\" (UID: \"e89365a9-1412-4108-b989-4fed047ed479\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs7dt" Oct 30 12:53:11.897322 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.897297 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg644\" (UniqueName: \"kubernetes.io/projected/fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a-kube-api-access-tg644\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbrmqw\" (UID: \"fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbrmqw" Oct 30 12:53:11.897360 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.897349 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e89365a9-1412-4108-b989-4fed047ed479-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs7dt\" (UID: \"e89365a9-1412-4108-b989-4fed047ed479\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs7dt" Oct 30 12:53:11.897393 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.897373 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbrmqw\" (UID: \"fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbrmqw" Oct 30 12:53:11.897489 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.897470 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e89365a9-1412-4108-b989-4fed047ed479-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs7dt\" (UID: \"e89365a9-1412-4108-b989-4fed047ed479\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs7dt" Oct 30 12:53:11.897653 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.897636 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e89365a9-1412-4108-b989-4fed047ed479-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs7dt\" (UID: \"e89365a9-1412-4108-b989-4fed047ed479\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs7dt" Oct 30 12:53:11.922285 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.922241 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s25qz\" (UniqueName: \"kubernetes.io/projected/e89365a9-1412-4108-b989-4fed047ed479-kube-api-access-s25qz\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs7dt\" (UID: \"e89365a9-1412-4108-b989-4fed047ed479\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs7dt" Oct 30 12:53:11.972476 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.972398 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs7dt" Oct 30 12:53:11.998612 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.998582 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbrmqw\" (UID: \"fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbrmqw" Oct 30 12:53:11.998714 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.998619 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbrmqw\" (UID: \"fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbrmqw" Oct 30 12:53:11.998714 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.998692 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg644\" (UniqueName: \"kubernetes.io/projected/fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a-kube-api-access-tg644\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbrmqw\" (UID: \"fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbrmqw" Oct 30 12:53:11.999006 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.998987 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbrmqw\" (UID: \"fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbrmqw" Oct 30 12:53:11.999046 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:11.998987 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbrmqw\" (UID: \"fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbrmqw" Oct 30 12:53:12.024485 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:12.024183 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg644\" (UniqueName: \"kubernetes.io/projected/fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a-kube-api-access-tg644\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbrmqw\" (UID: \"fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbrmqw" Oct 30 12:53:12.069153 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:12.069124 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbrmqw" Oct 30 12:53:12.082113 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:12.082087 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z97zc"] Oct 30 12:53:12.084886 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:53:12.084859 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod151643ed_3090_4391_a8ab_53243381fb2a.slice/crio-8c54b88c576b162ab2a4035979bc03c69ed64cd1f07dc692b5b427357fbab17b WatchSource:0}: Error finding container 8c54b88c576b162ab2a4035979bc03c69ed64cd1f07dc692b5b427357fbab17b: Status 404 returned error can't find the container with id 8c54b88c576b162ab2a4035979bc03c69ed64cd1f07dc692b5b427357fbab17b Oct 30 12:53:12.182087 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:12.182009 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7jgp"] Oct 30 12:53:12.184869 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:53:12.184840 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba1b363_bee4_4589_b486_135b70fa04a4.slice/crio-2f2c78e7d455a9530871ca17d23b1b4ffea91024650000d90be8f90557d140f2 WatchSource:0}: Error finding container 2f2c78e7d455a9530871ca17d23b1b4ffea91024650000d90be8f90557d140f2: Status 404 returned error can't find the container with id 2f2c78e7d455a9530871ca17d23b1b4ffea91024650000d90be8f90557d140f2 Oct 30 12:53:12.275041 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:12.275008 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs7dt"] Oct 30 12:53:12.282847 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:53:12.282820 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode89365a9_1412_4108_b989_4fed047ed479.slice/crio-9ddddcfb7163d450b8b4288b82f5fcc2af56d3479f6135ee14d00b1071f14498 WatchSource:0}: Error finding container 9ddddcfb7163d450b8b4288b82f5fcc2af56d3479f6135ee14d00b1071f14498: Status 404 returned error can't find the container with id 9ddddcfb7163d450b8b4288b82f5fcc2af56d3479f6135ee14d00b1071f14498 Oct 30 12:53:12.374452 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:12.374426 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbrmqw"] Oct 30 12:53:12.418119 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:53:12.418090 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe5ca43d_2371_4fc0_9a2d_f19ff7a95c3a.slice/crio-55ae691dbb205de59d21123342309a2ce58eac9eee083c8ab569cd592c45d507 WatchSource:0}: Error finding container 55ae691dbb205de59d21123342309a2ce58eac9eee083c8ab569cd592c45d507: Status 404 returned error can't find the container with id 55ae691dbb205de59d21123342309a2ce58eac9eee083c8ab569cd592c45d507 Oct 30 12:53:12.568363 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:12.568327 4931 generic.go:356] "Generic (PLEG): container finished" podID="fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a" containerID="c248168a7e0cb8955eb058183d4e00c0d31bc759dda35530c5e5963401e21117" exitCode=0 Oct 30 12:53:12.568788 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:12.568415 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbrmqw" event={"ID":"fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a","Type":"ContainerDied","Data":"c248168a7e0cb8955eb058183d4e00c0d31bc759dda35530c5e5963401e21117"} Oct 30 12:53:12.568788 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:12.568501 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbrmqw" event={"ID":"fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a","Type":"ContainerStarted","Data":"55ae691dbb205de59d21123342309a2ce58eac9eee083c8ab569cd592c45d507"} Oct 30 12:53:12.569859 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:12.569834 4931 generic.go:356] "Generic (PLEG): container finished" podID="4ba1b363-bee4-4589-b486-135b70fa04a4" containerID="9711f6dc0295041cce0c2d3df82060e1c7f253954219ec8d74ecc88748895ee8" exitCode=0 Oct 30 12:53:12.569982 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:12.569934 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7jgp" event={"ID":"4ba1b363-bee4-4589-b486-135b70fa04a4","Type":"ContainerDied","Data":"9711f6dc0295041cce0c2d3df82060e1c7f253954219ec8d74ecc88748895ee8"} Oct 30 12:53:12.570057 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:12.569981 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7jgp" event={"ID":"4ba1b363-bee4-4589-b486-135b70fa04a4","Type":"ContainerStarted","Data":"2f2c78e7d455a9530871ca17d23b1b4ffea91024650000d90be8f90557d140f2"} Oct 30 12:53:12.572215 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:12.572195 4931 generic.go:356] "Generic (PLEG): container finished" podID="e89365a9-1412-4108-b989-4fed047ed479" containerID="7306ef1dc446e97ee0cddbc2c208431128877a9b93c979ee32e3603cba04e444" exitCode=0 Oct 30 12:53:12.572340 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:12.572292 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs7dt" event={"ID":"e89365a9-1412-4108-b989-4fed047ed479","Type":"ContainerDied","Data":"7306ef1dc446e97ee0cddbc2c208431128877a9b93c979ee32e3603cba04e444"} Oct 30 12:53:12.572340 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:12.572330 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs7dt" event={"ID":"e89365a9-1412-4108-b989-4fed047ed479","Type":"ContainerStarted","Data":"9ddddcfb7163d450b8b4288b82f5fcc2af56d3479f6135ee14d00b1071f14498"} Oct 30 12:53:12.574117 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:12.574097 4931 generic.go:356] "Generic (PLEG): container finished" podID="151643ed-3090-4391-a8ab-53243381fb2a" containerID="3253b51b3379b6739dea066a66c4e9a20dbdbf591f4881d1c6923cf41e0a283a" exitCode=0 Oct 30 12:53:12.574218 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:12.574146 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z97zc" event={"ID":"151643ed-3090-4391-a8ab-53243381fb2a","Type":"ContainerDied","Data":"3253b51b3379b6739dea066a66c4e9a20dbdbf591f4881d1c6923cf41e0a283a"} Oct 30 12:53:12.574218 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:12.574174 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z97zc" event={"ID":"151643ed-3090-4391-a8ab-53243381fb2a","Type":"ContainerStarted","Data":"8c54b88c576b162ab2a4035979bc03c69ed64cd1f07dc692b5b427357fbab17b"} Oct 30 12:53:13.580043 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:13.579963 4931 generic.go:356] "Generic (PLEG): container finished" podID="151643ed-3090-4391-a8ab-53243381fb2a" containerID="4b702aede47f2acaa5c34a05c24720d4acfe12cb4bf4bba8b6eef13374b5c8e8" exitCode=0 Oct 30 12:53:13.580521 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:13.580037 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z97zc" event={"ID":"151643ed-3090-4391-a8ab-53243381fb2a","Type":"ContainerDied","Data":"4b702aede47f2acaa5c34a05c24720d4acfe12cb4bf4bba8b6eef13374b5c8e8"} Oct 30 12:53:13.581999 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:13.581969 4931 generic.go:356] "Generic (PLEG): container finished" podID="fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a" containerID="cf819acedd5391988db68b0afdd10e88f924e0fef5d111f9e8b572452ee35d92" exitCode=0 Oct 30 12:53:13.582077 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:13.582027 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbrmqw" event={"ID":"fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a","Type":"ContainerDied","Data":"cf819acedd5391988db68b0afdd10e88f924e0fef5d111f9e8b572452ee35d92"} Oct 30 12:53:13.584819 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:13.584729 4931 generic.go:356] "Generic (PLEG): container finished" podID="4ba1b363-bee4-4589-b486-135b70fa04a4" containerID="0b18ab3a62c47b0c879d0accc61faa3ec46d96b7e4d95b09f169c14f695c36e3" exitCode=0 Oct 30 12:53:13.584952 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:13.584832 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7jgp" event={"ID":"4ba1b363-bee4-4589-b486-135b70fa04a4","Type":"ContainerDied","Data":"0b18ab3a62c47b0c879d0accc61faa3ec46d96b7e4d95b09f169c14f695c36e3"} Oct 30 12:53:13.586747 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:13.586730 4931 generic.go:356] "Generic (PLEG): container finished" podID="e89365a9-1412-4108-b989-4fed047ed479" containerID="32a59b42b3230471efeb8fab24554550ab84c84d693d0e0123fd632da5f56e03" exitCode=0 Oct 30 12:53:13.586828 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:13.586766 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs7dt" event={"ID":"e89365a9-1412-4108-b989-4fed047ed479","Type":"ContainerDied","Data":"32a59b42b3230471efeb8fab24554550ab84c84d693d0e0123fd632da5f56e03"} Oct 30 12:53:14.594200 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:14.594167 4931 generic.go:356] "Generic (PLEG): container finished" podID="151643ed-3090-4391-a8ab-53243381fb2a" containerID="f3478687d3b2eb338f0bdf2b1fa9e1d07d902e309a1e47a9105ebcfcae1bed4c" exitCode=0 Oct 30 12:53:14.594628 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:14.594253 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z97zc" event={"ID":"151643ed-3090-4391-a8ab-53243381fb2a","Type":"ContainerDied","Data":"f3478687d3b2eb338f0bdf2b1fa9e1d07d902e309a1e47a9105ebcfcae1bed4c"} Oct 30 12:53:14.596343 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:14.596321 4931 generic.go:356] "Generic (PLEG): container finished" podID="fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a" containerID="7f2bb29f1eaa3ad08cbd9c35aaddef8d884136328c39c31ffb04210c5f0814b8" exitCode=0 Oct 30 12:53:14.596455 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:14.596386 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbrmqw" event={"ID":"fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a","Type":"ContainerDied","Data":"7f2bb29f1eaa3ad08cbd9c35aaddef8d884136328c39c31ffb04210c5f0814b8"} Oct 30 12:53:14.598320 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:14.598300 4931 generic.go:356] "Generic (PLEG): container finished" podID="4ba1b363-bee4-4589-b486-135b70fa04a4" containerID="13b592164e437bc150c961d110a442a9fcd3f9445f5c5957ea1c452b6e5114a0" exitCode=0 Oct 30 12:53:14.598415 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:14.598378 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7jgp" event={"ID":"4ba1b363-bee4-4589-b486-135b70fa04a4","Type":"ContainerDied","Data":"13b592164e437bc150c961d110a442a9fcd3f9445f5c5957ea1c452b6e5114a0"} Oct 30 12:53:14.600346 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:14.600327 4931 generic.go:356] "Generic (PLEG): container finished" podID="e89365a9-1412-4108-b989-4fed047ed479" containerID="238ab4314b889916b112dd7d0802becf3d7d77dd901fe8d73564fef64178ac88" exitCode=0 Oct 30 12:53:14.600429 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:14.600360 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs7dt" event={"ID":"e89365a9-1412-4108-b989-4fed047ed479","Type":"ContainerDied","Data":"238ab4314b889916b112dd7d0802becf3d7d77dd901fe8d73564fef64178ac88"} Oct 30 12:53:15.756966 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.756935 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs7dt" Oct 30 12:53:15.819338 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.819308 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7jgp" Oct 30 12:53:15.823495 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.823471 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z97zc" Oct 30 12:53:15.826832 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.826812 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbrmqw" Oct 30 12:53:15.837171 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.837150 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151643ed-3090-4391-a8ab-53243381fb2a-util\" (UniqueName: \"kubernetes.io/empty-dir/151643ed-3090-4391-a8ab-53243381fb2a-util\") pod \"151643ed-3090-4391-a8ab-53243381fb2a\" (UID: \"151643ed-3090-4391-a8ab-53243381fb2a\") " Oct 30 12:53:15.837254 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.837186 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ba1b363-bee4-4589-b486-135b70fa04a4-util\" (UniqueName: \"kubernetes.io/empty-dir/4ba1b363-bee4-4589-b486-135b70fa04a4-util\") pod \"4ba1b363-bee4-4589-b486-135b70fa04a4\" (UID: \"4ba1b363-bee4-4589-b486-135b70fa04a4\") " Oct 30 12:53:15.837254 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.837218 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a-bundle\") pod \"fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a\" (UID: \"fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a\") " Oct 30 12:53:15.837459 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.837276 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ba1b363-bee4-4589-b486-135b70fa04a4-kube-api-access-ktsvv\" (UniqueName: \"kubernetes.io/projected/4ba1b363-bee4-4589-b486-135b70fa04a4-kube-api-access-ktsvv\") pod \"4ba1b363-bee4-4589-b486-135b70fa04a4\" (UID: \"4ba1b363-bee4-4589-b486-135b70fa04a4\") " Oct 30 12:53:15.837459 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.837332 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151643ed-3090-4391-a8ab-53243381fb2a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/151643ed-3090-4391-a8ab-53243381fb2a-bundle\") pod \"151643ed-3090-4391-a8ab-53243381fb2a\" (UID: \"151643ed-3090-4391-a8ab-53243381fb2a\") " Oct 30 12:53:15.837459 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.837363 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e89365a9-1412-4108-b989-4fed047ed479-kube-api-access-s25qz\" (UniqueName: \"kubernetes.io/projected/e89365a9-1412-4108-b989-4fed047ed479-kube-api-access-s25qz\") pod \"e89365a9-1412-4108-b989-4fed047ed479\" (UID: \"e89365a9-1412-4108-b989-4fed047ed479\") " Oct 30 12:53:15.837459 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.837392 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ba1b363-bee4-4589-b486-135b70fa04a4-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ba1b363-bee4-4589-b486-135b70fa04a4-bundle\") pod \"4ba1b363-bee4-4589-b486-135b70fa04a4\" (UID: \"4ba1b363-bee4-4589-b486-135b70fa04a4\") " Oct 30 12:53:15.837459 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.837437 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e89365a9-1412-4108-b989-4fed047ed479-util\" (UniqueName: \"kubernetes.io/empty-dir/e89365a9-1412-4108-b989-4fed047ed479-util\") pod \"e89365a9-1412-4108-b989-4fed047ed479\" (UID: \"e89365a9-1412-4108-b989-4fed047ed479\") " Oct 30 12:53:15.837703 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.837471 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a-util\" (UniqueName: \"kubernetes.io/empty-dir/fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a-util\") pod \"fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a\" (UID: \"fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a\") " Oct 30 12:53:15.837703 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.837504 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/151643ed-3090-4391-a8ab-53243381fb2a-kube-api-access-6qvlf\" (UniqueName: \"kubernetes.io/projected/151643ed-3090-4391-a8ab-53243381fb2a-kube-api-access-6qvlf\") pod \"151643ed-3090-4391-a8ab-53243381fb2a\" (UID: \"151643ed-3090-4391-a8ab-53243381fb2a\") " Oct 30 12:53:15.837703 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.837530 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a-kube-api-access-tg644\" (UniqueName: \"kubernetes.io/projected/fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a-kube-api-access-tg644\") pod \"fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a\" (UID: \"fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a\") " Oct 30 12:53:15.837703 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.837558 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e89365a9-1412-4108-b989-4fed047ed479-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e89365a9-1412-4108-b989-4fed047ed479-bundle\") pod \"e89365a9-1412-4108-b989-4fed047ed479\" (UID: \"e89365a9-1412-4108-b989-4fed047ed479\") " Oct 30 12:53:15.838150 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.838116 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a-bundle" pod "fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a" (UID: "fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:53:15.838475 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.838449 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e89365a9-1412-4108-b989-4fed047ed479-bundle" pod "e89365a9-1412-4108-b989-4fed047ed479" (UID: "e89365a9-1412-4108-b989-4fed047ed479"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:53:15.839924 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.839525 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151643ed-3090-4391-a8ab-53243381fb2a-bundle" pod "151643ed-3090-4391-a8ab-53243381fb2a" (UID: "151643ed-3090-4391-a8ab-53243381fb2a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:53:15.840027 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.839975 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba1b363-bee4-4589-b486-135b70fa04a4-bundle" pod "4ba1b363-bee4-4589-b486-135b70fa04a4" (UID: "4ba1b363-bee4-4589-b486-135b70fa04a4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:53:15.840307 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.840255 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e89365a9-1412-4108-b989-4fed047ed479-kube-api-access-s25qz" pod "e89365a9-1412-4108-b989-4fed047ed479" (UID: "e89365a9-1412-4108-b989-4fed047ed479"). InnerVolumeSpecName "kube-api-access-s25qz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:53:15.841662 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.841630 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba1b363-bee4-4589-b486-135b70fa04a4-kube-api-access-ktsvv" pod "4ba1b363-bee4-4589-b486-135b70fa04a4" (UID: "4ba1b363-bee4-4589-b486-135b70fa04a4"). InnerVolumeSpecName "kube-api-access-ktsvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:53:15.842049 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.842024 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/151643ed-3090-4391-a8ab-53243381fb2a-kube-api-access-6qvlf" pod "151643ed-3090-4391-a8ab-53243381fb2a" (UID: "151643ed-3090-4391-a8ab-53243381fb2a"). InnerVolumeSpecName "kube-api-access-6qvlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:53:15.842256 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.842236 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a-kube-api-access-tg644" pod "fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a" (UID: "fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a"). InnerVolumeSpecName "kube-api-access-tg644". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:53:15.844678 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.844649 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151643ed-3090-4391-a8ab-53243381fb2a-util" pod "151643ed-3090-4391-a8ab-53243381fb2a" (UID: "151643ed-3090-4391-a8ab-53243381fb2a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:53:15.844837 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.844816 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba1b363-bee4-4589-b486-135b70fa04a4-util" pod "4ba1b363-bee4-4589-b486-135b70fa04a4" (UID: "4ba1b363-bee4-4589-b486-135b70fa04a4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:53:15.847635 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.847607 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e89365a9-1412-4108-b989-4fed047ed479-util" pod "e89365a9-1412-4108-b989-4fed047ed479" (UID: "e89365a9-1412-4108-b989-4fed047ed479"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:53:15.848166 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.848142 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a-util" pod "fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a" (UID: "fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:53:15.938938 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.938865 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qvlf\" (UniqueName: \"kubernetes.io/projected/151643ed-3090-4391-a8ab-53243381fb2a-kube-api-access-6qvlf\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:53:15.938938 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.938892 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg644\" (UniqueName: \"kubernetes.io/projected/fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a-kube-api-access-tg644\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:53:15.938938 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.938902 4931 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e89365a9-1412-4108-b989-4fed047ed479-bundle\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:53:15.938938 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.938915 4931 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/151643ed-3090-4391-a8ab-53243381fb2a-util\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:53:15.938938 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.938923 4931 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ba1b363-bee4-4589-b486-135b70fa04a4-util\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:53:15.938938 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.938931 4931 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a-bundle\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:53:15.938938 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.938938 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktsvv\" (UniqueName: \"kubernetes.io/projected/4ba1b363-bee4-4589-b486-135b70fa04a4-kube-api-access-ktsvv\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:53:15.939258 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.938947 4931 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/151643ed-3090-4391-a8ab-53243381fb2a-bundle\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:53:15.939258 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.938955 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s25qz\" (UniqueName: \"kubernetes.io/projected/e89365a9-1412-4108-b989-4fed047ed479-kube-api-access-s25qz\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:53:15.939258 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.938963 4931 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ba1b363-bee4-4589-b486-135b70fa04a4-bundle\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:53:15.939258 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.938971 4931 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e89365a9-1412-4108-b989-4fed047ed479-util\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:53:15.939258 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:15.938980 4931 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a-util\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:53:16.611510 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:16.611476 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs7dt" Oct 30 12:53:16.611704 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:16.611482 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs7dt" event={"ID":"e89365a9-1412-4108-b989-4fed047ed479","Type":"ContainerDied","Data":"9ddddcfb7163d450b8b4288b82f5fcc2af56d3479f6135ee14d00b1071f14498"} Oct 30 12:53:16.611704 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:16.611608 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ddddcfb7163d450b8b4288b82f5fcc2af56d3479f6135ee14d00b1071f14498" Oct 30 12:53:16.613419 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:16.613401 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z97zc" Oct 30 12:53:16.613561 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:16.613435 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z97zc" event={"ID":"151643ed-3090-4391-a8ab-53243381fb2a","Type":"ContainerDied","Data":"8c54b88c576b162ab2a4035979bc03c69ed64cd1f07dc692b5b427357fbab17b"} Oct 30 12:53:16.613561 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:16.613474 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c54b88c576b162ab2a4035979bc03c69ed64cd1f07dc692b5b427357fbab17b" Oct 30 12:53:16.615342 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:16.615323 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbrmqw" event={"ID":"fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a","Type":"ContainerDied","Data":"55ae691dbb205de59d21123342309a2ce58eac9eee083c8ab569cd592c45d507"} Oct 30 12:53:16.615451 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:16.615348 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbrmqw" Oct 30 12:53:16.615451 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:16.615351 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55ae691dbb205de59d21123342309a2ce58eac9eee083c8ab569cd592c45d507" Oct 30 12:53:16.617328 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:16.617307 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7jgp" event={"ID":"4ba1b363-bee4-4589-b486-135b70fa04a4","Type":"ContainerDied","Data":"2f2c78e7d455a9530871ca17d23b1b4ffea91024650000d90be8f90557d140f2"} Oct 30 12:53:16.617424 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:16.617339 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7jgp" Oct 30 12:53:16.617424 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:16.617344 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f2c78e7d455a9530871ca17d23b1b4ffea91024650000d90be8f90557d140f2" Oct 30 12:53:37.351335 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:53:37.351242 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 12:53:39.790790 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.790755 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-9x5sm"] Oct 30 12:53:39.791175 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.791079 4931 cpu_manager.go:414] "containerMap: removing container" podUID="fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a" containerName="util" Oct 30 12:53:39.791175 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.791090 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a" containerName="util" Oct 30 12:53:39.791175 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.791098 4931 cpu_manager.go:414] "containerMap: removing container" podUID="151643ed-3090-4391-a8ab-53243381fb2a" containerName="extract" Oct 30 12:53:39.791175 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.791102 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="151643ed-3090-4391-a8ab-53243381fb2a" containerName="extract" Oct 30 12:53:39.791175 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.791112 4931 cpu_manager.go:414] "containerMap: removing container" podUID="e89365a9-1412-4108-b989-4fed047ed479" containerName="pull" Oct 30 12:53:39.791175 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.791117 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e89365a9-1412-4108-b989-4fed047ed479" containerName="pull" Oct 30 12:53:39.791175 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.791125 4931 cpu_manager.go:414] "containerMap: removing container" podUID="e89365a9-1412-4108-b989-4fed047ed479" containerName="extract" Oct 30 12:53:39.791175 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.791130 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e89365a9-1412-4108-b989-4fed047ed479" containerName="extract" Oct 30 12:53:39.791175 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.791137 4931 cpu_manager.go:414] "containerMap: removing container" podUID="4ba1b363-bee4-4589-b486-135b70fa04a4" containerName="extract" Oct 30 12:53:39.791175 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.791142 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ba1b363-bee4-4589-b486-135b70fa04a4" containerName="extract" Oct 30 12:53:39.791175 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.791147 4931 cpu_manager.go:414] "containerMap: removing container" podUID="fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a" containerName="extract" Oct 30 12:53:39.791175 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.791160 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a" containerName="extract" Oct 30 12:53:39.791175 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.791180 4931 cpu_manager.go:414] "containerMap: removing container" podUID="4ba1b363-bee4-4589-b486-135b70fa04a4" containerName="pull" Oct 30 12:53:39.791602 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.791185 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ba1b363-bee4-4589-b486-135b70fa04a4" containerName="pull" Oct 30 12:53:39.791602 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.791196 4931 cpu_manager.go:414] "containerMap: removing container" podUID="4ba1b363-bee4-4589-b486-135b70fa04a4" containerName="util" Oct 30 12:53:39.791602 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.791201 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ba1b363-bee4-4589-b486-135b70fa04a4" containerName="util" Oct 30 12:53:39.791602 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.791209 4931 cpu_manager.go:414] "containerMap: removing container" podUID="fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a" containerName="pull" Oct 30 12:53:39.791602 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.791213 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a" containerName="pull" Oct 30 12:53:39.791602 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.791220 4931 cpu_manager.go:414] "containerMap: removing container" podUID="e89365a9-1412-4108-b989-4fed047ed479" containerName="util" Oct 30 12:53:39.791602 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.791224 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e89365a9-1412-4108-b989-4fed047ed479" containerName="util" Oct 30 12:53:39.791602 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.791232 4931 cpu_manager.go:414] "containerMap: removing container" podUID="151643ed-3090-4391-a8ab-53243381fb2a" containerName="pull" Oct 30 12:53:39.791602 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.791236 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="151643ed-3090-4391-a8ab-53243381fb2a" containerName="pull" Oct 30 12:53:39.791602 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.791243 4931 cpu_manager.go:414] "containerMap: removing container" podUID="151643ed-3090-4391-a8ab-53243381fb2a" containerName="util" Oct 30 12:53:39.791602 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.791247 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="151643ed-3090-4391-a8ab-53243381fb2a" containerName="util" Oct 30 12:53:39.791602 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.791312 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="151643ed-3090-4391-a8ab-53243381fb2a" containerName="extract" Oct 30 12:53:39.791602 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.791318 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="e89365a9-1412-4108-b989-4fed047ed479" containerName="extract" Oct 30 12:53:39.791602 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.791325 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ba1b363-bee4-4589-b486-135b70fa04a4" containerName="extract" Oct 30 12:53:39.791602 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.791331 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe5ca43d-2371-4fc0-9a2d-f19ff7a95c3a" containerName="extract" Oct 30 12:53:39.796062 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.796036 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9x5sm" Oct 30 12:53:39.816625 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.816592 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-9x5sm"] Oct 30 12:53:39.817545 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.817513 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 12:53:39.827689 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.827667 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 12:53:39.827689 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.827667 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 12:53:39.827827 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.827675 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 12:53:39.830885 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.830864 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-s2c7q\"" Oct 30 12:53:39.940321 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.940284 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8cl7\" (UniqueName: \"kubernetes.io/projected/f06f5bb4-b9de-4364-92b8-49a6c90125a3-kube-api-access-j8cl7\") pod \"kuadrant-console-plugin-66c4fb575d-9x5sm\" (UID: \"f06f5bb4-b9de-4364-92b8-49a6c90125a3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9x5sm" Oct 30 12:53:39.940505 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.940331 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f06f5bb4-b9de-4364-92b8-49a6c90125a3-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-9x5sm\" (UID: \"f06f5bb4-b9de-4364-92b8-49a6c90125a3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9x5sm" Oct 30 12:53:39.940505 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:39.940374 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f06f5bb4-b9de-4364-92b8-49a6c90125a3-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-9x5sm\" (UID: \"f06f5bb4-b9de-4364-92b8-49a6c90125a3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9x5sm" Oct 30 12:53:40.041115 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:40.041018 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8cl7\" (UniqueName: \"kubernetes.io/projected/f06f5bb4-b9de-4364-92b8-49a6c90125a3-kube-api-access-j8cl7\") pod \"kuadrant-console-plugin-66c4fb575d-9x5sm\" (UID: \"f06f5bb4-b9de-4364-92b8-49a6c90125a3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9x5sm" Oct 30 12:53:40.041115 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:40.041073 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f06f5bb4-b9de-4364-92b8-49a6c90125a3-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-9x5sm\" (UID: \"f06f5bb4-b9de-4364-92b8-49a6c90125a3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9x5sm" Oct 30 12:53:40.041376 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:40.041133 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f06f5bb4-b9de-4364-92b8-49a6c90125a3-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-9x5sm\" (UID: \"f06f5bb4-b9de-4364-92b8-49a6c90125a3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9x5sm" Oct 30 12:53:40.041883 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:40.041822 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f06f5bb4-b9de-4364-92b8-49a6c90125a3-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-9x5sm\" (UID: \"f06f5bb4-b9de-4364-92b8-49a6c90125a3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9x5sm" Oct 30 12:53:40.043534 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:40.043510 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f06f5bb4-b9de-4364-92b8-49a6c90125a3-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-9x5sm\" (UID: \"f06f5bb4-b9de-4364-92b8-49a6c90125a3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9x5sm" Oct 30 12:53:40.066412 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:40.066377 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8cl7\" (UniqueName: \"kubernetes.io/projected/f06f5bb4-b9de-4364-92b8-49a6c90125a3-kube-api-access-j8cl7\") pod \"kuadrant-console-plugin-66c4fb575d-9x5sm\" (UID: \"f06f5bb4-b9de-4364-92b8-49a6c90125a3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9x5sm" Oct 30 12:53:40.107340 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:40.107308 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9x5sm" Oct 30 12:53:40.405051 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:40.405022 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-9x5sm"] Oct 30 12:53:40.407321 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:53:40.407286 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf06f5bb4_b9de_4364_92b8_49a6c90125a3.slice/crio-02287e1fe9e248a782cb0e171b08a56a3470bf1c2946af720ec5c2e9664edc67 WatchSource:0}: Error finding container 02287e1fe9e248a782cb0e171b08a56a3470bf1c2946af720ec5c2e9664edc67: Status 404 returned error can't find the container with id 02287e1fe9e248a782cb0e171b08a56a3470bf1c2946af720ec5c2e9664edc67 Oct 30 12:53:40.733872 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:40.733834 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9x5sm" event={"ID":"f06f5bb4-b9de-4364-92b8-49a6c90125a3","Type":"ContainerStarted","Data":"02287e1fe9e248a782cb0e171b08a56a3470bf1c2946af720ec5c2e9664edc67"} Oct 30 12:53:45.761778 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:45.761741 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9x5sm" event={"ID":"f06f5bb4-b9de-4364-92b8-49a6c90125a3","Type":"ContainerStarted","Data":"f137708db517455e4b7b45847b0e4daf6432d34914987d614d2dc415d9db7a47"} Oct 30 12:53:45.812694 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:45.812643 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9x5sm" podStartSLOduration=2.344436636 podStartE2EDuration="6.812628665s" totalImagesPullingTime="4.468192029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:45.811981324 +0000 UTC m=+524.415479771" watchObservedRunningTime="2026-10-30 12:53:45.812628665 +0000 UTC m=+524.416127111" Oct 30 12:53:48.961536 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:48.961500 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jhdz7"] Oct 30 12:53:48.966497 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:48.966473 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jhdz7" Oct 30 12:53:48.976332 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:48.976304 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jhdz7"] Oct 30 12:53:48.987644 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:48.987611 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-k8kf6\"" Oct 30 12:53:49.124365 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:49.124320 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kld4w\" (UniqueName: \"kubernetes.io/projected/ffa8f391-3406-4582-96d6-a9a591041067-kube-api-access-kld4w\") pod \"kuadrant-operator-controller-manager-6b76784d48-jhdz7\" (UID: \"ffa8f391-3406-4582-96d6-a9a591041067\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jhdz7" Oct 30 12:53:49.124365 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:49.124383 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffa8f391-3406-4582-96d6-a9a591041067-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-jhdz7\" (UID: \"ffa8f391-3406-4582-96d6-a9a591041067\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jhdz7" Oct 30 12:53:49.225725 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:49.225640 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kld4w\" (UniqueName: \"kubernetes.io/projected/ffa8f391-3406-4582-96d6-a9a591041067-kube-api-access-kld4w\") pod \"kuadrant-operator-controller-manager-6b76784d48-jhdz7\" (UID: \"ffa8f391-3406-4582-96d6-a9a591041067\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jhdz7" Oct 30 12:53:49.225725 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:49.225687 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffa8f391-3406-4582-96d6-a9a591041067-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-jhdz7\" (UID: \"ffa8f391-3406-4582-96d6-a9a591041067\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jhdz7" Oct 30 12:53:49.226029 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:49.226011 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffa8f391-3406-4582-96d6-a9a591041067-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-jhdz7\" (UID: \"ffa8f391-3406-4582-96d6-a9a591041067\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jhdz7" Oct 30 12:53:49.250491 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:49.250465 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kld4w\" (UniqueName: \"kubernetes.io/projected/ffa8f391-3406-4582-96d6-a9a591041067-kube-api-access-kld4w\") pod \"kuadrant-operator-controller-manager-6b76784d48-jhdz7\" (UID: \"ffa8f391-3406-4582-96d6-a9a591041067\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jhdz7" Oct 30 12:53:49.281377 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:49.281355 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jhdz7" Oct 30 12:53:49.609288 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:49.609237 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jhdz7"] Oct 30 12:53:49.611563 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:53:49.611540 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffa8f391_3406_4582_96d6_a9a591041067.slice/crio-938384f3fb47a91efcca042e6286b93ad7fa38196fc858b715b22c16fcf2a433 WatchSource:0}: Error finding container 938384f3fb47a91efcca042e6286b93ad7fa38196fc858b715b22c16fcf2a433: Status 404 returned error can't find the container with id 938384f3fb47a91efcca042e6286b93ad7fa38196fc858b715b22c16fcf2a433 Oct 30 12:53:49.784261 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:49.784216 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jhdz7" event={"ID":"ffa8f391-3406-4582-96d6-a9a591041067","Type":"ContainerStarted","Data":"938384f3fb47a91efcca042e6286b93ad7fa38196fc858b715b22c16fcf2a433"} Oct 30 12:53:51.342846 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:51.342812 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jhdz7"] Oct 30 12:53:51.345285 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:51.345233 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jhdz7"] Oct 30 12:53:51.359744 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:51.359719 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dwwt7"] Oct 30 12:53:51.373663 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:51.373626 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dwwt7" Oct 30 12:53:51.391059 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:51.391033 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dwwt7"] Oct 30 12:53:51.547016 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:51.546979 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cfrm\" (UniqueName: \"kubernetes.io/projected/02614ddc-69f5-464f-b260-113dbbf46bbc-kube-api-access-4cfrm\") pod \"kuadrant-operator-controller-manager-6b76784d48-dwwt7\" (UID: \"02614ddc-69f5-464f-b260-113dbbf46bbc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dwwt7" Oct 30 12:53:51.547182 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:51.547072 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/02614ddc-69f5-464f-b260-113dbbf46bbc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-dwwt7\" (UID: \"02614ddc-69f5-464f-b260-113dbbf46bbc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dwwt7" Oct 30 12:53:51.648067 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:51.647982 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cfrm\" (UniqueName: \"kubernetes.io/projected/02614ddc-69f5-464f-b260-113dbbf46bbc-kube-api-access-4cfrm\") pod \"kuadrant-operator-controller-manager-6b76784d48-dwwt7\" (UID: \"02614ddc-69f5-464f-b260-113dbbf46bbc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dwwt7" Oct 30 12:53:51.648244 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:51.648094 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/02614ddc-69f5-464f-b260-113dbbf46bbc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-dwwt7\" (UID: \"02614ddc-69f5-464f-b260-113dbbf46bbc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dwwt7" Oct 30 12:53:51.648539 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:51.648512 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/02614ddc-69f5-464f-b260-113dbbf46bbc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-dwwt7\" (UID: \"02614ddc-69f5-464f-b260-113dbbf46bbc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dwwt7" Oct 30 12:53:51.673284 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:51.673235 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cfrm\" (UniqueName: \"kubernetes.io/projected/02614ddc-69f5-464f-b260-113dbbf46bbc-kube-api-access-4cfrm\") pod \"kuadrant-operator-controller-manager-6b76784d48-dwwt7\" (UID: \"02614ddc-69f5-464f-b260-113dbbf46bbc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dwwt7" Oct 30 12:53:51.709305 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:51.709254 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dwwt7" Oct 30 12:53:52.050165 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:52.050134 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dwwt7"] Oct 30 12:53:52.052866 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:53:52.052833 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02614ddc_69f5_464f_b260_113dbbf46bbc.slice/crio-5cde05265d1a8cdb80ec675cb88c8de61544ff642d91fe7a5c08a1f65905a2d9 WatchSource:0}: Error finding container 5cde05265d1a8cdb80ec675cb88c8de61544ff642d91fe7a5c08a1f65905a2d9: Status 404 returned error can't find the container with id 5cde05265d1a8cdb80ec675cb88c8de61544ff642d91fe7a5c08a1f65905a2d9 Oct 30 12:53:52.804894 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:52.804847 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dwwt7" event={"ID":"02614ddc-69f5-464f-b260-113dbbf46bbc","Type":"ContainerStarted","Data":"5cde05265d1a8cdb80ec675cb88c8de61544ff642d91fe7a5c08a1f65905a2d9"} Oct 30 12:53:53.812301 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:53.812246 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jhdz7" podUID="ffa8f391-3406-4582-96d6-a9a591041067" containerName="manager" containerID="cri-o://b168d4acc3f4d88e51b5d1279f398c77e900bad00d000913e59598d9dea3a9f6" gracePeriod=2 Oct 30 12:53:54.090311 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:54.090283 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jhdz7" Oct 30 12:53:54.105122 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:53:54.105089 4931 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-jhdz7\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" podUID="ffa8f391-3406-4582-96d6-a9a591041067" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jhdz7" Oct 30 12:53:54.270931 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:54.270872 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffa8f391-3406-4582-96d6-a9a591041067-kube-api-access-kld4w\" (UniqueName: \"kubernetes.io/projected/ffa8f391-3406-4582-96d6-a9a591041067-kube-api-access-kld4w\") pod \"ffa8f391-3406-4582-96d6-a9a591041067\" (UID: \"ffa8f391-3406-4582-96d6-a9a591041067\") " Oct 30 12:53:54.270931 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:54.270906 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffa8f391-3406-4582-96d6-a9a591041067-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffa8f391-3406-4582-96d6-a9a591041067-extensions-socket-volume\") pod \"ffa8f391-3406-4582-96d6-a9a591041067\" (UID: \"ffa8f391-3406-4582-96d6-a9a591041067\") " Oct 30 12:53:54.271214 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:54.271186 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffa8f391-3406-4582-96d6-a9a591041067-extensions-socket-volume" pod "ffa8f391-3406-4582-96d6-a9a591041067" (UID: "ffa8f391-3406-4582-96d6-a9a591041067"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:53:54.272842 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:54.272821 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffa8f391-3406-4582-96d6-a9a591041067-kube-api-access-kld4w" pod "ffa8f391-3406-4582-96d6-a9a591041067" (UID: "ffa8f391-3406-4582-96d6-a9a591041067"). InnerVolumeSpecName "kube-api-access-kld4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:53:54.371733 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:54.371655 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kld4w\" (UniqueName: \"kubernetes.io/projected/ffa8f391-3406-4582-96d6-a9a591041067-kube-api-access-kld4w\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:53:54.371733 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:54.371689 4931 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffa8f391-3406-4582-96d6-a9a591041067-extensions-socket-volume\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:53:54.818855 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:54.818825 4931 generic.go:356] "Generic (PLEG): container finished" podID="ffa8f391-3406-4582-96d6-a9a591041067" containerID="b168d4acc3f4d88e51b5d1279f398c77e900bad00d000913e59598d9dea3a9f6" exitCode=2 Oct 30 12:53:54.819367 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:54.818869 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jhdz7" Oct 30 12:53:54.819367 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:54.818909 4931 scope.go:122] "RemoveContainer" containerID="b168d4acc3f4d88e51b5d1279f398c77e900bad00d000913e59598d9dea3a9f6" Oct 30 12:53:54.820815 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:54.820786 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dwwt7" event={"ID":"02614ddc-69f5-464f-b260-113dbbf46bbc","Type":"ContainerStarted","Data":"0119959c31c92fe0be0e457c805175d9efaba25f0e2e7dbb22f59f35653bc99b"} Oct 30 12:53:54.820910 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:54.820854 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dwwt7" Oct 30 12:53:54.828072 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:54.828055 4931 scope.go:122] "RemoveContainer" containerID="b168d4acc3f4d88e51b5d1279f398c77e900bad00d000913e59598d9dea3a9f6" Oct 30 12:53:54.828343 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:53:54.828322 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b168d4acc3f4d88e51b5d1279f398c77e900bad00d000913e59598d9dea3a9f6\": container with ID starting with b168d4acc3f4d88e51b5d1279f398c77e900bad00d000913e59598d9dea3a9f6 not found: ID does not exist" containerID="b168d4acc3f4d88e51b5d1279f398c77e900bad00d000913e59598d9dea3a9f6" Oct 30 12:53:54.828394 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:54.828351 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b168d4acc3f4d88e51b5d1279f398c77e900bad00d000913e59598d9dea3a9f6"} err="failed to get container status \"b168d4acc3f4d88e51b5d1279f398c77e900bad00d000913e59598d9dea3a9f6\": rpc error: code = NotFound desc = could not find container \"b168d4acc3f4d88e51b5d1279f398c77e900bad00d000913e59598d9dea3a9f6\": container with ID starting with b168d4acc3f4d88e51b5d1279f398c77e900bad00d000913e59598d9dea3a9f6 not found: ID does not exist" Oct 30 12:53:54.833898 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:53:54.833867 4931 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-jhdz7\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" podUID="ffa8f391-3406-4582-96d6-a9a591041067" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jhdz7" Oct 30 12:53:54.848573 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:53:54.848545 4931 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-jhdz7\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" podUID="ffa8f391-3406-4582-96d6-a9a591041067" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jhdz7" Oct 30 12:53:54.921893 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:54.921849 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dwwt7" podStartSLOduration=1.7597836070000001 podStartE2EDuration="3.921836027s" totalImagesPullingTime="2.16205242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:54.907261877 +0000 UTC m=+533.510760323" watchObservedRunningTime="2026-10-30 12:53:54.921836027 +0000 UTC m=+533.525334472" Oct 30 12:53:55.511559 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:53:55.511528 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffa8f391-3406-4582-96d6-a9a591041067" path="/var/lib/kubelet/pods/ffa8f391-3406-4582-96d6-a9a591041067/volumes" Oct 30 12:54:05.829663 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:05.829632 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dwwt7" Oct 30 12:54:07.350390 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:54:07.350352 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 12:54:18.700650 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:18.700612 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dwwt7"] Oct 30 12:54:18.701224 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:18.700925 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dwwt7" podUID="02614ddc-69f5-464f-b260-113dbbf46bbc" containerName="manager" containerID="cri-o://0119959c31c92fe0be0e457c805175d9efaba25f0e2e7dbb22f59f35653bc99b" gracePeriod=10 Oct 30 12:54:18.938678 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:18.938648 4931 generic.go:356] "Generic (PLEG): container finished" podID="02614ddc-69f5-464f-b260-113dbbf46bbc" containerID="0119959c31c92fe0be0e457c805175d9efaba25f0e2e7dbb22f59f35653bc99b" exitCode=0 Oct 30 12:54:18.938825 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:18.938717 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dwwt7" event={"ID":"02614ddc-69f5-464f-b260-113dbbf46bbc","Type":"ContainerDied","Data":"0119959c31c92fe0be0e457c805175d9efaba25f0e2e7dbb22f59f35653bc99b"} Oct 30 12:54:18.953867 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:18.953812 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dwwt7" Oct 30 12:54:18.972913 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:18.972889 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02614ddc-69f5-464f-b260-113dbbf46bbc-kube-api-access-4cfrm\" (UniqueName: \"kubernetes.io/projected/02614ddc-69f5-464f-b260-113dbbf46bbc-kube-api-access-4cfrm\") pod \"02614ddc-69f5-464f-b260-113dbbf46bbc\" (UID: \"02614ddc-69f5-464f-b260-113dbbf46bbc\") " Oct 30 12:54:18.973028 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:18.972949 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02614ddc-69f5-464f-b260-113dbbf46bbc-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/02614ddc-69f5-464f-b260-113dbbf46bbc-extensions-socket-volume\") pod \"02614ddc-69f5-464f-b260-113dbbf46bbc\" (UID: \"02614ddc-69f5-464f-b260-113dbbf46bbc\") " Oct 30 12:54:18.973373 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:18.973351 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02614ddc-69f5-464f-b260-113dbbf46bbc-extensions-socket-volume" pod "02614ddc-69f5-464f-b260-113dbbf46bbc" (UID: "02614ddc-69f5-464f-b260-113dbbf46bbc"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:54:18.974935 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:18.974913 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02614ddc-69f5-464f-b260-113dbbf46bbc-kube-api-access-4cfrm" pod "02614ddc-69f5-464f-b260-113dbbf46bbc" (UID: "02614ddc-69f5-464f-b260-113dbbf46bbc"). InnerVolumeSpecName "kube-api-access-4cfrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:54:19.073636 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:19.073605 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cfrm\" (UniqueName: \"kubernetes.io/projected/02614ddc-69f5-464f-b260-113dbbf46bbc-kube-api-access-4cfrm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:54:19.073636 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:19.073632 4931 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/02614ddc-69f5-464f-b260-113dbbf46bbc-extensions-socket-volume\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:54:19.945399 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:19.945375 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dwwt7" Oct 30 12:54:19.945873 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:19.945375 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dwwt7" event={"ID":"02614ddc-69f5-464f-b260-113dbbf46bbc","Type":"ContainerDied","Data":"5cde05265d1a8cdb80ec675cb88c8de61544ff642d91fe7a5c08a1f65905a2d9"} Oct 30 12:54:19.945873 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:19.945523 4931 scope.go:122] "RemoveContainer" containerID="0119959c31c92fe0be0e457c805175d9efaba25f0e2e7dbb22f59f35653bc99b" Oct 30 12:54:20.017787 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:20.017756 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dwwt7"] Oct 30 12:54:20.031300 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:20.031259 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dwwt7"] Oct 30 12:54:21.511585 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:21.511552 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02614ddc-69f5-464f-b260-113dbbf46bbc" path="/var/lib/kubelet/pods/02614ddc-69f5-464f-b260-113dbbf46bbc/volumes" Oct 30 12:54:23.223156 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.223119 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx"] Oct 30 12:54:23.223557 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.223481 4931 cpu_manager.go:414] "containerMap: removing container" podUID="02614ddc-69f5-464f-b260-113dbbf46bbc" containerName="manager" Oct 30 12:54:23.223557 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.223493 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02614ddc-69f5-464f-b260-113dbbf46bbc" containerName="manager" Oct 30 12:54:23.223557 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.223508 4931 cpu_manager.go:414] "containerMap: removing container" podUID="ffa8f391-3406-4582-96d6-a9a591041067" containerName="manager" Oct 30 12:54:23.223557 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.223514 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffa8f391-3406-4582-96d6-a9a591041067" containerName="manager" Oct 30 12:54:23.223714 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.223587 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="02614ddc-69f5-464f-b260-113dbbf46bbc" containerName="manager" Oct 30 12:54:23.223714 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.223595 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffa8f391-3406-4582-96d6-a9a591041067" containerName="manager" Oct 30 12:54:23.226717 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.226696 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:54:23.245895 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.245873 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4s79t\"" Oct 30 12:54:23.260727 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.260704 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx"] Oct 30 12:54:23.312072 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.312046 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d126cf3a-50cd-45e4-b37f-b268fdbb3d13-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-ddspx\" (UID: \"d126cf3a-50cd-45e4-b37f-b268fdbb3d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:54:23.312072 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.312075 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d126cf3a-50cd-45e4-b37f-b268fdbb3d13-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-ddspx\" (UID: \"d126cf3a-50cd-45e4-b37f-b268fdbb3d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:54:23.312288 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.312093 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf89m\" (UniqueName: \"kubernetes.io/projected/d126cf3a-50cd-45e4-b37f-b268fdbb3d13-kube-api-access-wf89m\") pod \"maas-default-gateway-openshift-default-58c6585447-ddspx\" (UID: \"d126cf3a-50cd-45e4-b37f-b268fdbb3d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:54:23.312288 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.312163 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d126cf3a-50cd-45e4-b37f-b268fdbb3d13-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-ddspx\" (UID: \"d126cf3a-50cd-45e4-b37f-b268fdbb3d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:54:23.312288 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.312207 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d126cf3a-50cd-45e4-b37f-b268fdbb3d13-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-ddspx\" (UID: \"d126cf3a-50cd-45e4-b37f-b268fdbb3d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:54:23.312288 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.312228 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d126cf3a-50cd-45e4-b37f-b268fdbb3d13-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-ddspx\" (UID: \"d126cf3a-50cd-45e4-b37f-b268fdbb3d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:54:23.312288 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.312243 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d126cf3a-50cd-45e4-b37f-b268fdbb3d13-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-ddspx\" (UID: \"d126cf3a-50cd-45e4-b37f-b268fdbb3d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:54:23.312461 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.312430 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d126cf3a-50cd-45e4-b37f-b268fdbb3d13-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-ddspx\" (UID: \"d126cf3a-50cd-45e4-b37f-b268fdbb3d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:54:23.312495 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.312464 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d126cf3a-50cd-45e4-b37f-b268fdbb3d13-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-ddspx\" (UID: \"d126cf3a-50cd-45e4-b37f-b268fdbb3d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:54:23.413651 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.413618 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf89m\" (UniqueName: \"kubernetes.io/projected/d126cf3a-50cd-45e4-b37f-b268fdbb3d13-kube-api-access-wf89m\") pod \"maas-default-gateway-openshift-default-58c6585447-ddspx\" (UID: \"d126cf3a-50cd-45e4-b37f-b268fdbb3d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:54:23.413827 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.413660 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d126cf3a-50cd-45e4-b37f-b268fdbb3d13-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-ddspx\" (UID: \"d126cf3a-50cd-45e4-b37f-b268fdbb3d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:54:23.413827 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.413685 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d126cf3a-50cd-45e4-b37f-b268fdbb3d13-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-ddspx\" (UID: \"d126cf3a-50cd-45e4-b37f-b268fdbb3d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:54:23.413827 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.413716 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d126cf3a-50cd-45e4-b37f-b268fdbb3d13-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-ddspx\" (UID: \"d126cf3a-50cd-45e4-b37f-b268fdbb3d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:54:23.413827 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.413739 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d126cf3a-50cd-45e4-b37f-b268fdbb3d13-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-ddspx\" (UID: \"d126cf3a-50cd-45e4-b37f-b268fdbb3d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:54:23.414042 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.413959 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d126cf3a-50cd-45e4-b37f-b268fdbb3d13-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-ddspx\" (UID: \"d126cf3a-50cd-45e4-b37f-b268fdbb3d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:54:23.414042 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.414029 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d126cf3a-50cd-45e4-b37f-b268fdbb3d13-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-ddspx\" (UID: \"d126cf3a-50cd-45e4-b37f-b268fdbb3d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:54:23.414156 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.414089 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d126cf3a-50cd-45e4-b37f-b268fdbb3d13-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-ddspx\" (UID: \"d126cf3a-50cd-45e4-b37f-b268fdbb3d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:54:23.414156 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.414113 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d126cf3a-50cd-45e4-b37f-b268fdbb3d13-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-ddspx\" (UID: \"d126cf3a-50cd-45e4-b37f-b268fdbb3d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:54:23.414297 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.414256 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d126cf3a-50cd-45e4-b37f-b268fdbb3d13-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-ddspx\" (UID: \"d126cf3a-50cd-45e4-b37f-b268fdbb3d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:54:23.414405 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.414382 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d126cf3a-50cd-45e4-b37f-b268fdbb3d13-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-ddspx\" (UID: \"d126cf3a-50cd-45e4-b37f-b268fdbb3d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:54:23.414467 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.414429 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d126cf3a-50cd-45e4-b37f-b268fdbb3d13-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-ddspx\" (UID: \"d126cf3a-50cd-45e4-b37f-b268fdbb3d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:54:23.414563 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.414540 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d126cf3a-50cd-45e4-b37f-b268fdbb3d13-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-ddspx\" (UID: \"d126cf3a-50cd-45e4-b37f-b268fdbb3d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:54:23.414628 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.414609 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d126cf3a-50cd-45e4-b37f-b268fdbb3d13-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-ddspx\" (UID: \"d126cf3a-50cd-45e4-b37f-b268fdbb3d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:54:23.416079 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.416057 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d126cf3a-50cd-45e4-b37f-b268fdbb3d13-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-ddspx\" (UID: \"d126cf3a-50cd-45e4-b37f-b268fdbb3d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:54:23.416636 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.416615 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d126cf3a-50cd-45e4-b37f-b268fdbb3d13-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-ddspx\" (UID: \"d126cf3a-50cd-45e4-b37f-b268fdbb3d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:54:23.442223 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.442195 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf89m\" (UniqueName: \"kubernetes.io/projected/d126cf3a-50cd-45e4-b37f-b268fdbb3d13-kube-api-access-wf89m\") pod \"maas-default-gateway-openshift-default-58c6585447-ddspx\" (UID: \"d126cf3a-50cd-45e4-b37f-b268fdbb3d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:54:23.442916 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.442888 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d126cf3a-50cd-45e4-b37f-b268fdbb3d13-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-ddspx\" (UID: \"d126cf3a-50cd-45e4-b37f-b268fdbb3d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:54:23.539239 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.539171 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:54:23.854999 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.854972 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx"] Oct 30 12:54:23.856429 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:54:23.856403 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd126cf3a_50cd_45e4_b37f_b268fdbb3d13.slice/crio-91a3d819aba929cdd38c0e0dcc9342a7d93ea423a3046904f078d72d6e89e09f WatchSource:0}: Error finding container 91a3d819aba929cdd38c0e0dcc9342a7d93ea423a3046904f078d72d6e89e09f: Status 404 returned error can't find the container with id 91a3d819aba929cdd38c0e0dcc9342a7d93ea423a3046904f078d72d6e89e09f Oct 30 12:54:23.858367 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.858334 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:54:23.858451 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.858394 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:54:23.858451 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.858424 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:54:23.967921 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.967881 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" event={"ID":"d126cf3a-50cd-45e4-b37f-b268fdbb3d13","Type":"ContainerStarted","Data":"fe1418d6f188b7c2d00c17cb449d1211d6f244742d681bc57bdbf911e4509843"} Oct 30 12:54:23.968100 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:23.967950 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" event={"ID":"d126cf3a-50cd-45e4-b37f-b268fdbb3d13","Type":"ContainerStarted","Data":"91a3d819aba929cdd38c0e0dcc9342a7d93ea423a3046904f078d72d6e89e09f"} Oct 30 12:54:24.016668 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:24.016613 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" podStartSLOduration=1.01659548 podStartE2EDuration="1.01659548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:24.016462109 +0000 UTC m=+562.619960556" watchObservedRunningTime="2026-10-30 12:54:24.01659548 +0000 UTC m=+562.620093928" Oct 30 12:54:24.539526 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:24.539486 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:54:24.545121 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:24.545089 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:54:24.973824 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:24.973780 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:54:24.974692 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:24.974673 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:54:37.351725 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:54:37.351621 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 12:54:40.893152 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:40.893116 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 12:54:40.896539 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:40.896524 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-hb5z9" Oct 30 12:54:40.935206 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:40.935179 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 12:54:40.937299 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:40.937260 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 12:54:40.963238 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:40.963210 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99cbg\" (UniqueName: \"kubernetes.io/projected/b6fd15bb-fccd-4bf2-b86c-6b7dc183af33-kube-api-access-99cbg\") pod \"limitador-limitador-7c9b87d574-hb5z9\" (UID: \"b6fd15bb-fccd-4bf2-b86c-6b7dc183af33\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-hb5z9" Oct 30 12:54:40.963405 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:40.963386 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b6fd15bb-fccd-4bf2-b86c-6b7dc183af33-config-file\") pod \"limitador-limitador-7c9b87d574-hb5z9\" (UID: \"b6fd15bb-fccd-4bf2-b86c-6b7dc183af33\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-hb5z9" Oct 30 12:54:40.980709 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:40.980682 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 12:54:41.064781 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:41.064746 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b6fd15bb-fccd-4bf2-b86c-6b7dc183af33-config-file\") pod \"limitador-limitador-7c9b87d574-hb5z9\" (UID: \"b6fd15bb-fccd-4bf2-b86c-6b7dc183af33\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-hb5z9" Oct 30 12:54:41.064949 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:41.064805 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99cbg\" (UniqueName: \"kubernetes.io/projected/b6fd15bb-fccd-4bf2-b86c-6b7dc183af33-kube-api-access-99cbg\") pod \"limitador-limitador-7c9b87d574-hb5z9\" (UID: \"b6fd15bb-fccd-4bf2-b86c-6b7dc183af33\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-hb5z9" Oct 30 12:54:41.065472 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:41.065453 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b6fd15bb-fccd-4bf2-b86c-6b7dc183af33-config-file\") pod \"limitador-limitador-7c9b87d574-hb5z9\" (UID: \"b6fd15bb-fccd-4bf2-b86c-6b7dc183af33\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-hb5z9" Oct 30 12:54:41.088985 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:41.088956 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99cbg\" (UniqueName: \"kubernetes.io/projected/b6fd15bb-fccd-4bf2-b86c-6b7dc183af33-kube-api-access-99cbg\") pod \"limitador-limitador-7c9b87d574-hb5z9\" (UID: \"b6fd15bb-fccd-4bf2-b86c-6b7dc183af33\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-hb5z9" Oct 30 12:54:41.208890 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:41.208846 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-hb5z9" Oct 30 12:54:41.510903 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:54:41.510872 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6fd15bb_fccd_4bf2_b86c_6b7dc183af33.slice/crio-5ed212329896207cec88e4a198170e8a31cee5a955539b25621eb1fff3208d4a WatchSource:0}: Error finding container 5ed212329896207cec88e4a198170e8a31cee5a955539b25621eb1fff3208d4a: Status 404 returned error can't find the container with id 5ed212329896207cec88e4a198170e8a31cee5a955539b25621eb1fff3208d4a Oct 30 12:54:41.512383 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:41.512363 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 12:54:42.067014 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:42.066948 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-hb5z9" event={"ID":"b6fd15bb-fccd-4bf2-b86c-6b7dc183af33","Type":"ContainerStarted","Data":"5ed212329896207cec88e4a198170e8a31cee5a955539b25621eb1fff3208d4a"} Oct 30 12:54:43.073155 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:43.073080 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-hb5z9" event={"ID":"b6fd15bb-fccd-4bf2-b86c-6b7dc183af33","Type":"ContainerStarted","Data":"8b007a04b61073717e855f862582c3781e6998273e46ee8110e55608e308f010"} Oct 30 12:54:43.073557 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:43.073215 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-hb5z9" Oct 30 12:54:43.116323 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:43.116258 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-hb5z9" podStartSLOduration=2.279657314 podStartE2EDuration="3.116244008s" totalImagesPullingTime="836.586694ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:43.115811692 +0000 UTC m=+581.719310119" watchObservedRunningTime="2026-10-30 12:54:43.116244008 +0000 UTC m=+581.719742453" Oct 30 12:54:54.079587 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:54:54.079496 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-hb5z9" Oct 30 12:55:01.558885 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:55:01.558851 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/2.log" Oct 30 12:55:01.559373 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:55:01.559033 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/2.log" Oct 30 12:55:07.352645 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:55:07.352603 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 12:55:37.351942 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:55:37.351902 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 12:56:02.288955 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:02.288921 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 12:56:07.375589 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:56:07.375552 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 12:56:11.016488 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:11.016447 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-ddspx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:56:11.018236 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:11.016521 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" podUID="d126cf3a-50cd-45e4-b37f-b268fdbb3d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:56:11.501335 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:11.501300 4931 generic.go:356] "Generic (PLEG): container finished" podID="d126cf3a-50cd-45e4-b37f-b268fdbb3d13" containerID="fe1418d6f188b7c2d00c17cb449d1211d6f244742d681bc57bdbf911e4509843" exitCode=0 Oct 30 12:56:11.501533 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:11.501373 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" event={"ID":"d126cf3a-50cd-45e4-b37f-b268fdbb3d13","Type":"ContainerDied","Data":"fe1418d6f188b7c2d00c17cb449d1211d6f244742d681bc57bdbf911e4509843"} Oct 30 12:56:11.501790 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:11.501774 4931 scope.go:122] "RemoveContainer" containerID="fe1418d6f188b7c2d00c17cb449d1211d6f244742d681bc57bdbf911e4509843" Oct 30 12:56:11.502974 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:11.502945 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:56:11.503041 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:11.503015 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:56:11.503075 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:11.503053 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:56:12.353291 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:12.353147 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-44c8c"] Oct 30 12:56:12.361804 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:12.361774 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-44c8c" Oct 30 12:56:12.371573 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:12.371540 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-44c8c"] Oct 30 12:56:12.457292 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:12.456745 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2677f24-72a2-4130-b530-4720923f33a8-cert\") pod \"llmisvc-controller-manager-6466c554d8-44c8c\" (UID: \"f2677f24-72a2-4130-b530-4720923f33a8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-44c8c" Oct 30 12:56:12.457292 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:12.456804 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvg9v\" (UniqueName: \"kubernetes.io/projected/f2677f24-72a2-4130-b530-4720923f33a8-kube-api-access-hvg9v\") pod \"llmisvc-controller-manager-6466c554d8-44c8c\" (UID: \"f2677f24-72a2-4130-b530-4720923f33a8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-44c8c" Oct 30 12:56:12.507808 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:12.507766 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" event={"ID":"d126cf3a-50cd-45e4-b37f-b268fdbb3d13","Type":"ContainerStarted","Data":"33a85d9b64c84ec9322e1dc9df4dc2776cd5ed843c789849bc6b282a3991652e"} Oct 30 12:56:12.552742 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:12.552656 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:56:12.557866 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:12.557827 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2677f24-72a2-4130-b530-4720923f33a8-cert\") pod \"llmisvc-controller-manager-6466c554d8-44c8c\" (UID: \"f2677f24-72a2-4130-b530-4720923f33a8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-44c8c" Oct 30 12:56:12.558071 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:12.557875 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvg9v\" (UniqueName: \"kubernetes.io/projected/f2677f24-72a2-4130-b530-4720923f33a8-kube-api-access-hvg9v\") pod \"llmisvc-controller-manager-6466c554d8-44c8c\" (UID: \"f2677f24-72a2-4130-b530-4720923f33a8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-44c8c" Oct 30 12:56:12.558253 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:12.558229 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-ddspx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 30 12:56:12.558367 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:12.558305 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" podUID="d126cf3a-50cd-45e4-b37f-b268fdbb3d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 30 12:56:12.560985 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:12.560956 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2677f24-72a2-4130-b530-4720923f33a8-cert\") pod \"llmisvc-controller-manager-6466c554d8-44c8c\" (UID: \"f2677f24-72a2-4130-b530-4720923f33a8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-44c8c" Oct 30 12:56:12.590427 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:12.590405 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvg9v\" (UniqueName: \"kubernetes.io/projected/f2677f24-72a2-4130-b530-4720923f33a8-kube-api-access-hvg9v\") pod \"llmisvc-controller-manager-6466c554d8-44c8c\" (UID: \"f2677f24-72a2-4130-b530-4720923f33a8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-44c8c" Oct 30 12:56:12.675799 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:12.675706 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-44c8c" Oct 30 12:56:12.981984 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:12.981961 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-44c8c"] Oct 30 12:56:12.984423 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:56:12.984394 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2677f24_72a2_4130_b530_4720923f33a8.slice/crio-a52dff96b7a4a1c05b03793a3a2e6dd11a8c91d834561274e61961339277c22e WatchSource:0}: Error finding container a52dff96b7a4a1c05b03793a3a2e6dd11a8c91d834561274e61961339277c22e: Status 404 returned error can't find the container with id a52dff96b7a4a1c05b03793a3a2e6dd11a8c91d834561274e61961339277c22e Oct 30 12:56:12.986206 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:12.986187 4931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:56:13.513181 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:13.513147 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-44c8c" event={"ID":"f2677f24-72a2-4130-b530-4720923f33a8","Type":"ContainerStarted","Data":"a52dff96b7a4a1c05b03793a3a2e6dd11a8c91d834561274e61961339277c22e"} Oct 30 12:56:13.557735 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:13.557689 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-ddspx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 30 12:56:13.557899 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:13.557778 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" podUID="d126cf3a-50cd-45e4-b37f-b268fdbb3d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 30 12:56:14.519549 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:14.519516 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-44c8c" event={"ID":"f2677f24-72a2-4130-b530-4720923f33a8","Type":"ContainerStarted","Data":"102408b797e639243acdc46be916a7fef9b1da5a57faa467ba607f71eefe3526"} Oct 30 12:56:14.519953 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:14.519642 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-44c8c" Oct 30 12:56:14.557404 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:14.557366 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-ddspx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 30 12:56:14.557619 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:14.557420 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" podUID="d126cf3a-50cd-45e4-b37f-b268fdbb3d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 30 12:56:14.567861 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:14.567805 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-44c8c" podStartSLOduration=1.970585583 podStartE2EDuration="2.567790537s" totalImagesPullingTime="597.204954ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:14.567158053 +0000 UTC m=+673.170656489" watchObservedRunningTime="2026-10-30 12:56:14.567790537 +0000 UTC m=+673.171288977" Oct 30 12:56:15.558002 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:15.557957 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-ddspx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 30 12:56:15.558556 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:15.558032 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" podUID="d126cf3a-50cd-45e4-b37f-b268fdbb3d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 30 12:56:16.554289 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:16.551679 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-ddspx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 30 12:56:16.554289 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:16.551740 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" podUID="d126cf3a-50cd-45e4-b37f-b268fdbb3d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 30 12:56:17.547322 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:17.547287 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:56:17.547811 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:17.547702 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:56:17.548579 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:17.548557 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" Oct 30 12:56:37.351693 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:56:37.351650 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 12:56:45.526822 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:45.526793 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-44c8c" Oct 30 12:56:45.622386 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:45.622349 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-c6n2t"] Oct 30 12:56:45.622638 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:45.622607 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-c6n2t" podUID="2dc4e69c-b092-4f23-99f5-64b9436a5a6c" containerName="manager" containerID="cri-o://ec9796fc60a0c524c41eb746df0724902e7d19c6a4a3128a40e2950cc3937beb" gracePeriod=30 Oct 30 12:56:45.873138 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:45.873113 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-c6n2t" Oct 30 12:56:45.950647 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:45.950609 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dc4e69c-b092-4f23-99f5-64b9436a5a6c-kube-api-access-2zcgl\" (UniqueName: \"kubernetes.io/projected/2dc4e69c-b092-4f23-99f5-64b9436a5a6c-kube-api-access-2zcgl\") pod \"2dc4e69c-b092-4f23-99f5-64b9436a5a6c\" (UID: \"2dc4e69c-b092-4f23-99f5-64b9436a5a6c\") " Oct 30 12:56:45.950806 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:45.950700 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dc4e69c-b092-4f23-99f5-64b9436a5a6c-cert\" (UniqueName: \"kubernetes.io/secret/2dc4e69c-b092-4f23-99f5-64b9436a5a6c-cert\") pod \"2dc4e69c-b092-4f23-99f5-64b9436a5a6c\" (UID: \"2dc4e69c-b092-4f23-99f5-64b9436a5a6c\") " Oct 30 12:56:45.952753 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:45.952717 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dc4e69c-b092-4f23-99f5-64b9436a5a6c-cert" pod "2dc4e69c-b092-4f23-99f5-64b9436a5a6c" (UID: "2dc4e69c-b092-4f23-99f5-64b9436a5a6c"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:56:45.952874 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:45.952805 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc4e69c-b092-4f23-99f5-64b9436a5a6c-kube-api-access-2zcgl" pod "2dc4e69c-b092-4f23-99f5-64b9436a5a6c" (UID: "2dc4e69c-b092-4f23-99f5-64b9436a5a6c"). InnerVolumeSpecName "kube-api-access-2zcgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:56:46.052306 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:46.052213 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zcgl\" (UniqueName: \"kubernetes.io/projected/2dc4e69c-b092-4f23-99f5-64b9436a5a6c-kube-api-access-2zcgl\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:56:46.052306 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:46.052238 4931 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2dc4e69c-b092-4f23-99f5-64b9436a5a6c-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 12:56:46.680067 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:46.680036 4931 generic.go:356] "Generic (PLEG): container finished" podID="2dc4e69c-b092-4f23-99f5-64b9436a5a6c" containerID="ec9796fc60a0c524c41eb746df0724902e7d19c6a4a3128a40e2950cc3937beb" exitCode=0 Oct 30 12:56:46.680515 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:46.680082 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-c6n2t" Oct 30 12:56:46.680515 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:46.680111 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-c6n2t" event={"ID":"2dc4e69c-b092-4f23-99f5-64b9436a5a6c","Type":"ContainerDied","Data":"ec9796fc60a0c524c41eb746df0724902e7d19c6a4a3128a40e2950cc3937beb"} Oct 30 12:56:46.680515 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:46.680152 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-c6n2t" event={"ID":"2dc4e69c-b092-4f23-99f5-64b9436a5a6c","Type":"ContainerDied","Data":"c9804bb4ee3128714dd084dbd1e64e9e07a7d74da16cec492ccdf8cc53c5bf4f"} Oct 30 12:56:46.680515 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:46.680174 4931 scope.go:122] "RemoveContainer" containerID="ec9796fc60a0c524c41eb746df0724902e7d19c6a4a3128a40e2950cc3937beb" Oct 30 12:56:46.689536 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:46.689518 4931 scope.go:122] "RemoveContainer" containerID="ec9796fc60a0c524c41eb746df0724902e7d19c6a4a3128a40e2950cc3937beb" Oct 30 12:56:46.689804 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:56:46.689784 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec9796fc60a0c524c41eb746df0724902e7d19c6a4a3128a40e2950cc3937beb\": container with ID starting with ec9796fc60a0c524c41eb746df0724902e7d19c6a4a3128a40e2950cc3937beb not found: ID does not exist" containerID="ec9796fc60a0c524c41eb746df0724902e7d19c6a4a3128a40e2950cc3937beb" Oct 30 12:56:46.689868 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:46.689818 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec9796fc60a0c524c41eb746df0724902e7d19c6a4a3128a40e2950cc3937beb"} err="failed to get container status \"ec9796fc60a0c524c41eb746df0724902e7d19c6a4a3128a40e2950cc3937beb\": rpc error: code = NotFound desc = could not find container \"ec9796fc60a0c524c41eb746df0724902e7d19c6a4a3128a40e2950cc3937beb\": container with ID starting with ec9796fc60a0c524c41eb746df0724902e7d19c6a4a3128a40e2950cc3937beb not found: ID does not exist" Oct 30 12:56:46.736473 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:46.736442 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-c6n2t"] Oct 30 12:56:46.740463 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:46.740430 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-c6n2t"] Oct 30 12:56:47.511278 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:56:47.511233 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dc4e69c-b092-4f23-99f5-64b9436a5a6c" path="/var/lib/kubelet/pods/2dc4e69c-b092-4f23-99f5-64b9436a5a6c/volumes" Oct 30 12:57:07.351740 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:57:07.351704 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 12:57:07.625318 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:07.625196 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 12:57:09.894488 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:09.894451 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 12:57:16.605345 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:16.605311 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 12:57:22.334401 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:22.334365 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 12:57:37.351170 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:57:37.351131 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 12:57:39.277249 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.277213 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96"] Oct 30 12:57:39.277651 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.277620 4931 cpu_manager.go:414] "containerMap: removing container" podUID="2dc4e69c-b092-4f23-99f5-64b9436a5a6c" containerName="manager" Oct 30 12:57:39.277651 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.277631 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dc4e69c-b092-4f23-99f5-64b9436a5a6c" containerName="manager" Oct 30 12:57:39.277721 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.277715 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dc4e69c-b092-4f23-99f5-64b9436a5a6c" containerName="manager" Oct 30 12:57:39.280856 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.280839 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" Oct 30 12:57:39.310809 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.310777 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96"] Oct 30 12:57:39.408420 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.408387 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01524a5a-15e5-45ad-a9cb-15b6d605bf86-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-q8w96\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" Oct 30 12:57:39.408420 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.408425 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01524a5a-15e5-45ad-a9cb-15b6d605bf86-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-q8w96\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" Oct 30 12:57:39.408670 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.408447 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01524a5a-15e5-45ad-a9cb-15b6d605bf86-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-q8w96\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" Oct 30 12:57:39.408670 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.408586 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01524a5a-15e5-45ad-a9cb-15b6d605bf86-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-q8w96\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" Oct 30 12:57:39.408778 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.408669 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01524a5a-15e5-45ad-a9cb-15b6d605bf86-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-q8w96\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" Oct 30 12:57:39.408778 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.408697 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01524a5a-15e5-45ad-a9cb-15b6d605bf86-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-q8w96\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" Oct 30 12:57:39.408778 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.408724 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01524a5a-15e5-45ad-a9cb-15b6d605bf86-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-q8w96\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" Oct 30 12:57:39.408941 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.408821 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sgb2\" (UniqueName: \"kubernetes.io/projected/01524a5a-15e5-45ad-a9cb-15b6d605bf86-kube-api-access-5sgb2\") pod \"maas-default-gateway-openshift-default-58c6585447-q8w96\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" Oct 30 12:57:39.408941 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.408869 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01524a5a-15e5-45ad-a9cb-15b6d605bf86-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-q8w96\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" Oct 30 12:57:39.513378 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.513347 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01524a5a-15e5-45ad-a9cb-15b6d605bf86-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-q8w96\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" Oct 30 12:57:39.513556 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.513400 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sgb2\" (UniqueName: \"kubernetes.io/projected/01524a5a-15e5-45ad-a9cb-15b6d605bf86-kube-api-access-5sgb2\") pod \"maas-default-gateway-openshift-default-58c6585447-q8w96\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" Oct 30 12:57:39.513556 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.513436 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01524a5a-15e5-45ad-a9cb-15b6d605bf86-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-q8w96\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" Oct 30 12:57:39.513673 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.513581 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01524a5a-15e5-45ad-a9cb-15b6d605bf86-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-q8w96\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" Oct 30 12:57:39.513673 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.513638 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01524a5a-15e5-45ad-a9cb-15b6d605bf86-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-q8w96\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" Oct 30 12:57:39.513673 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.513664 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01524a5a-15e5-45ad-a9cb-15b6d605bf86-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-q8w96\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" Oct 30 12:57:39.513811 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.513703 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01524a5a-15e5-45ad-a9cb-15b6d605bf86-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-q8w96\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" Oct 30 12:57:39.513811 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.513763 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01524a5a-15e5-45ad-a9cb-15b6d605bf86-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-q8w96\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" Oct 30 12:57:39.513811 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.513795 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01524a5a-15e5-45ad-a9cb-15b6d605bf86-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-q8w96\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" Oct 30 12:57:39.514070 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.514045 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01524a5a-15e5-45ad-a9cb-15b6d605bf86-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-q8w96\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" Oct 30 12:57:39.514231 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.514121 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01524a5a-15e5-45ad-a9cb-15b6d605bf86-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-q8w96\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" Oct 30 12:57:39.514376 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.514171 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01524a5a-15e5-45ad-a9cb-15b6d605bf86-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-q8w96\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" Oct 30 12:57:39.514496 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.514463 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01524a5a-15e5-45ad-a9cb-15b6d605bf86-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-q8w96\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" Oct 30 12:57:39.514705 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.514682 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01524a5a-15e5-45ad-a9cb-15b6d605bf86-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-q8w96\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" Oct 30 12:57:39.516474 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.516449 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01524a5a-15e5-45ad-a9cb-15b6d605bf86-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-q8w96\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" Oct 30 12:57:39.516553 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.516534 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01524a5a-15e5-45ad-a9cb-15b6d605bf86-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-q8w96\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" Oct 30 12:57:39.538618 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.538567 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sgb2\" (UniqueName: \"kubernetes.io/projected/01524a5a-15e5-45ad-a9cb-15b6d605bf86-kube-api-access-5sgb2\") pod \"maas-default-gateway-openshift-default-58c6585447-q8w96\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" Oct 30 12:57:39.542128 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.542106 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01524a5a-15e5-45ad-a9cb-15b6d605bf86-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-q8w96\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" Oct 30 12:57:39.595236 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.595211 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" Oct 30 12:57:39.916068 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.915992 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96"] Oct 30 12:57:39.919020 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:57:39.918996 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01524a5a_15e5_45ad_a9cb_15b6d605bf86.slice/crio-bca7b90f276f189ad4c84f8c3fc9cd9a1dce282b6d325c10bc7dac6fa3a9198a WatchSource:0}: Error finding container bca7b90f276f189ad4c84f8c3fc9cd9a1dce282b6d325c10bc7dac6fa3a9198a: Status 404 returned error can't find the container with id bca7b90f276f189ad4c84f8c3fc9cd9a1dce282b6d325c10bc7dac6fa3a9198a Oct 30 12:57:39.920976 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.920944 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:57:39.921057 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.921005 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:57:39.921057 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.921034 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:57:39.970215 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:39.970189 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" event={"ID":"01524a5a-15e5-45ad-a9cb-15b6d605bf86","Type":"ContainerStarted","Data":"bca7b90f276f189ad4c84f8c3fc9cd9a1dce282b6d325c10bc7dac6fa3a9198a"} Oct 30 12:57:40.976475 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:40.976438 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" event={"ID":"01524a5a-15e5-45ad-a9cb-15b6d605bf86","Type":"ContainerStarted","Data":"cb52ab5cbeb1fe1eeb3aeaf4327093fe8c5f08fb1918fdfb545ce9aaac9c939a"} Oct 30 12:57:41.025838 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:41.025785 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" podStartSLOduration=2.025769088 podStartE2EDuration="2.025769088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:41.023818308 +0000 UTC m=+759.627316755" watchObservedRunningTime="2026-10-30 12:57:41.025769088 +0000 UTC m=+759.629267535" Oct 30 12:57:41.595652 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:41.595615 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" Oct 30 12:57:41.597144 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:41.597117 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-q8w96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 30 12:57:41.597287 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:41.597177 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" podUID="01524a5a-15e5-45ad-a9cb-15b6d605bf86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 30 12:57:42.596070 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:42.596036 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-q8w96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 30 12:57:42.596506 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:42.596089 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" podUID="01524a5a-15e5-45ad-a9cb-15b6d605bf86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 30 12:57:43.596073 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:43.596038 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-q8w96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 30 12:57:43.596480 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:43.596094 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" podUID="01524a5a-15e5-45ad-a9cb-15b6d605bf86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 30 12:57:44.596081 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:44.596041 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-q8w96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 30 12:57:44.596509 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:44.596107 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" podUID="01524a5a-15e5-45ad-a9cb-15b6d605bf86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 30 12:57:45.596590 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:45.596553 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-q8w96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 30 12:57:45.596968 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:45.596632 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" podUID="01524a5a-15e5-45ad-a9cb-15b6d605bf86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 30 12:57:46.596393 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:46.596352 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-q8w96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 30 12:57:46.596606 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:46.596421 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" podUID="01524a5a-15e5-45ad-a9cb-15b6d605bf86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 30 12:57:48.209473 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:48.209442 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" Oct 30 12:57:48.210356 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:48.209893 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" Oct 30 12:57:48.210675 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:48.210653 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" Oct 30 12:57:54.330751 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:54.330715 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w"] Oct 30 12:57:54.334317 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:54.334299 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" Oct 30 12:57:54.371064 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:54.371034 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w"] Oct 30 12:57:54.371354 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:54.371331 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dd90613-12ac-4509-9f2e-b5c1e199719f-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-tvm7w\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" Oct 30 12:57:54.371425 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:54.371373 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dd90613-12ac-4509-9f2e-b5c1e199719f-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-tvm7w\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" Oct 30 12:57:54.371463 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:54.371445 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dd90613-12ac-4509-9f2e-b5c1e199719f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-tvm7w\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" Oct 30 12:57:54.371496 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:54.371474 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dd90613-12ac-4509-9f2e-b5c1e199719f-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-tvm7w\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" Oct 30 12:57:54.371538 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:54.371504 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgfbv\" (UniqueName: \"kubernetes.io/projected/3dd90613-12ac-4509-9f2e-b5c1e199719f-kube-api-access-lgfbv\") pod \"maas-default-gateway-openshift-default-58c6585447-tvm7w\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" Oct 30 12:57:54.371574 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:54.371540 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dd90613-12ac-4509-9f2e-b5c1e199719f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-tvm7w\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" Oct 30 12:57:54.371609 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:54.371589 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dd90613-12ac-4509-9f2e-b5c1e199719f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-tvm7w\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" Oct 30 12:57:54.371681 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:54.371669 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dd90613-12ac-4509-9f2e-b5c1e199719f-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-tvm7w\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" Oct 30 12:57:54.371715 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:54.371703 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dd90613-12ac-4509-9f2e-b5c1e199719f-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-tvm7w\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" Oct 30 12:57:54.472385 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:54.472352 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dd90613-12ac-4509-9f2e-b5c1e199719f-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-tvm7w\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" Oct 30 12:57:54.472576 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:54.472398 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dd90613-12ac-4509-9f2e-b5c1e199719f-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-tvm7w\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" Oct 30 12:57:54.472576 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:54.472436 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dd90613-12ac-4509-9f2e-b5c1e199719f-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-tvm7w\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" Oct 30 12:57:54.472576 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:54.472476 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dd90613-12ac-4509-9f2e-b5c1e199719f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-tvm7w\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" Oct 30 12:57:54.472576 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:54.472510 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dd90613-12ac-4509-9f2e-b5c1e199719f-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-tvm7w\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" Oct 30 12:57:54.472576 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:54.472545 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgfbv\" (UniqueName: \"kubernetes.io/projected/3dd90613-12ac-4509-9f2e-b5c1e199719f-kube-api-access-lgfbv\") pod \"maas-default-gateway-openshift-default-58c6585447-tvm7w\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" Oct 30 12:57:54.472576 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:54.472574 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dd90613-12ac-4509-9f2e-b5c1e199719f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-tvm7w\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" Oct 30 12:57:54.472887 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:54.472604 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dd90613-12ac-4509-9f2e-b5c1e199719f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-tvm7w\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" Oct 30 12:57:54.472887 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:54.472633 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dd90613-12ac-4509-9f2e-b5c1e199719f-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-tvm7w\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" Oct 30 12:57:54.472887 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:54.472820 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dd90613-12ac-4509-9f2e-b5c1e199719f-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-tvm7w\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" Oct 30 12:57:54.473007 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:54.472882 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dd90613-12ac-4509-9f2e-b5c1e199719f-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-tvm7w\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" Oct 30 12:57:54.473007 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:54.472913 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dd90613-12ac-4509-9f2e-b5c1e199719f-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-tvm7w\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" Oct 30 12:57:54.473007 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:54.472970 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dd90613-12ac-4509-9f2e-b5c1e199719f-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-tvm7w\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" Oct 30 12:57:54.473348 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:54.473329 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dd90613-12ac-4509-9f2e-b5c1e199719f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-tvm7w\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" Oct 30 12:57:54.474831 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:54.474812 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dd90613-12ac-4509-9f2e-b5c1e199719f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-tvm7w\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" Oct 30 12:57:54.474907 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:54.474890 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dd90613-12ac-4509-9f2e-b5c1e199719f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-tvm7w\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" Oct 30 12:57:54.497744 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:54.497716 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgfbv\" (UniqueName: \"kubernetes.io/projected/3dd90613-12ac-4509-9f2e-b5c1e199719f-kube-api-access-lgfbv\") pod \"maas-default-gateway-openshift-default-58c6585447-tvm7w\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" Oct 30 12:57:54.497845 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:54.497774 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dd90613-12ac-4509-9f2e-b5c1e199719f-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-tvm7w\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" Oct 30 12:57:54.649586 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:54.649501 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" Oct 30 12:57:54.973855 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:54.973828 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w"] Oct 30 12:57:54.975480 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:57:54.975450 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dd90613_12ac_4509_9f2e_b5c1e199719f.slice/crio-97ce0758162b098bdddfc83ce422da0e6b2d6fd38f6c7ce19987153ce7e92988 WatchSource:0}: Error finding container 97ce0758162b098bdddfc83ce422da0e6b2d6fd38f6c7ce19987153ce7e92988: Status 404 returned error can't find the container with id 97ce0758162b098bdddfc83ce422da0e6b2d6fd38f6c7ce19987153ce7e92988 Oct 30 12:57:54.977525 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:54.977491 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:57:54.977623 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:54.977569 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:57:54.977623 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:54.977610 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:57:55.048500 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:55.048463 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" event={"ID":"3dd90613-12ac-4509-9f2e-b5c1e199719f","Type":"ContainerStarted","Data":"a3844c4b0fed5f26e5fdb8fa7c78ad4c6c54b52251504aca93960d762f3377d5"} Oct 30 12:57:55.048631 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:55.048535 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" event={"ID":"3dd90613-12ac-4509-9f2e-b5c1e199719f","Type":"ContainerStarted","Data":"97ce0758162b098bdddfc83ce422da0e6b2d6fd38f6c7ce19987153ce7e92988"} Oct 30 12:57:55.100554 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:55.100499 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" podStartSLOduration=1.100485254 podStartE2EDuration="1.100485254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:55.099538658 +0000 UTC m=+773.703037109" watchObservedRunningTime="2026-10-30 12:57:55.100485254 +0000 UTC m=+773.703983699" Oct 30 12:57:55.654290 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:55.653896 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" Oct 30 12:57:56.654292 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:56.653600 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-tvm7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 12:57:56.654292 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:56.653660 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" podUID="3dd90613-12ac-4509-9f2e-b5c1e199719f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 12:57:57.654290 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:57.651627 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-tvm7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 12:57:57.654290 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:57.651686 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" podUID="3dd90613-12ac-4509-9f2e-b5c1e199719f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 12:57:58.657293 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:58.654564 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-tvm7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 12:57:58.657293 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:58.654626 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" podUID="3dd90613-12ac-4509-9f2e-b5c1e199719f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 12:57:59.654290 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:59.653699 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-tvm7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 12:57:59.654290 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:57:59.653763 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" podUID="3dd90613-12ac-4509-9f2e-b5c1e199719f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 12:58:00.652291 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:00.651752 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-tvm7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 12:58:00.652291 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:00.651835 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" podUID="3dd90613-12ac-4509-9f2e-b5c1e199719f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 12:58:01.650260 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:01.650228 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-tvm7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 12:58:01.650461 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:01.650303 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" podUID="3dd90613-12ac-4509-9f2e-b5c1e199719f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 12:58:03.255167 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:03.255126 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" Oct 30 12:58:03.255632 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:03.255566 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" Oct 30 12:58:03.256368 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:03.256348 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" Oct 30 12:58:07.351153 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:58:07.351118 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 12:58:37.354416 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:58:37.354373 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 12:58:49.358841 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:49.358804 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb"] Oct 30 12:58:49.362379 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:49.362362 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb" Oct 30 12:58:49.385100 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:49.385069 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb"] Oct 30 12:58:49.386073 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:49.386050 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-dd90cb\"" Oct 30 12:58:49.395138 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:49.395115 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-dd90cb-dockercfg-m6xs4\"" Oct 30 12:58:49.461699 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:49.461660 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/440d993b-e62e-4284-955d-2e79adccec2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb\" (UID: \"440d993b-e62e-4284-955d-2e79adccec2e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb" Oct 30 12:58:49.461909 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:49.461815 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l2x5\" (UniqueName: \"kubernetes.io/projected/440d993b-e62e-4284-955d-2e79adccec2e-kube-api-access-4l2x5\") pod \"payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb\" (UID: \"440d993b-e62e-4284-955d-2e79adccec2e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb" Oct 30 12:58:49.562765 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:49.562729 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/440d993b-e62e-4284-955d-2e79adccec2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb\" (UID: \"440d993b-e62e-4284-955d-2e79adccec2e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb" Oct 30 12:58:49.562936 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:49.562799 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l2x5\" (UniqueName: \"kubernetes.io/projected/440d993b-e62e-4284-955d-2e79adccec2e-kube-api-access-4l2x5\") pod \"payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb\" (UID: \"440d993b-e62e-4284-955d-2e79adccec2e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb" Oct 30 12:58:49.563469 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:49.563450 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/440d993b-e62e-4284-955d-2e79adccec2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb\" (UID: \"440d993b-e62e-4284-955d-2e79adccec2e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb" Oct 30 12:58:49.590926 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:49.590887 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l2x5\" (UniqueName: \"kubernetes.io/projected/440d993b-e62e-4284-955d-2e79adccec2e-kube-api-access-4l2x5\") pod \"payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb\" (UID: \"440d993b-e62e-4284-955d-2e79adccec2e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb" Oct 30 12:58:49.675581 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:49.675498 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb" Oct 30 12:58:49.976932 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:49.976907 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb"] Oct 30 12:58:49.979060 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:58:49.979034 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440d993b_e62e_4284_955d_2e79adccec2e.slice/crio-f8dd40c4b2f79b0b132b5e8538d6d1e2b11aa3afc1073b049c5197e966cfd0f4 WatchSource:0}: Error finding container f8dd40c4b2f79b0b132b5e8538d6d1e2b11aa3afc1073b049c5197e966cfd0f4: Status 404 returned error can't find the container with id f8dd40c4b2f79b0b132b5e8538d6d1e2b11aa3afc1073b049c5197e966cfd0f4 Oct 30 12:58:50.342179 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:50.342098 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb" event={"ID":"440d993b-e62e-4284-955d-2e79adccec2e","Type":"ContainerStarted","Data":"f8dd40c4b2f79b0b132b5e8538d6d1e2b11aa3afc1073b049c5197e966cfd0f4"} Oct 30 12:58:50.822930 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:50.822898 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 12:58:51.174290 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:51.174180 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt"] Oct 30 12:58:51.181248 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:51.181218 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt" Oct 30 12:58:51.203334 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:51.203295 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt"] Oct 30 12:58:51.229992 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:51.229939 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-7a3603\"" Oct 30 12:58:51.259913 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:51.259874 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-7a3603-dockercfg-cntmd\"" Oct 30 12:58:51.280230 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:51.280191 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2tbr\" (UniqueName: \"kubernetes.io/projected/03309fcc-c2dd-4424-9deb-c34596cd1558-kube-api-access-b2tbr\") pod \"payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt\" (UID: \"03309fcc-c2dd-4424-9deb-c34596cd1558\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt" Oct 30 12:58:51.280415 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:51.280250 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03309fcc-c2dd-4424-9deb-c34596cd1558-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt\" (UID: \"03309fcc-c2dd-4424-9deb-c34596cd1558\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt" Oct 30 12:58:51.381928 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:51.381884 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2tbr\" (UniqueName: \"kubernetes.io/projected/03309fcc-c2dd-4424-9deb-c34596cd1558-kube-api-access-b2tbr\") pod \"payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt\" (UID: \"03309fcc-c2dd-4424-9deb-c34596cd1558\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt" Oct 30 12:58:51.382141 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:51.381979 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03309fcc-c2dd-4424-9deb-c34596cd1558-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt\" (UID: \"03309fcc-c2dd-4424-9deb-c34596cd1558\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt" Oct 30 12:58:51.382739 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:51.382716 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03309fcc-c2dd-4424-9deb-c34596cd1558-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt\" (UID: \"03309fcc-c2dd-4424-9deb-c34596cd1558\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt" Oct 30 12:58:51.412826 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:51.412793 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2tbr\" (UniqueName: \"kubernetes.io/projected/03309fcc-c2dd-4424-9deb-c34596cd1558-kube-api-access-b2tbr\") pod \"payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt\" (UID: \"03309fcc-c2dd-4424-9deb-c34596cd1558\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt" Oct 30 12:58:51.498014 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:51.497973 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt" Oct 30 12:58:51.820750 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:51.820725 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt"] Oct 30 12:58:51.824868 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:58:51.824836 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03309fcc_c2dd_4424_9deb_c34596cd1558.slice/crio-7c887ec035061014e2ccf3ddbeb41d6bc5d9d72dac92703c5f2a957ffeee2bd2 WatchSource:0}: Error finding container 7c887ec035061014e2ccf3ddbeb41d6bc5d9d72dac92703c5f2a957ffeee2bd2: Status 404 returned error can't find the container with id 7c887ec035061014e2ccf3ddbeb41d6bc5d9d72dac92703c5f2a957ffeee2bd2 Oct 30 12:58:52.242536 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:52.242501 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78"] Oct 30 12:58:52.247023 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:52.246951 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78" Oct 30 12:58:52.264712 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:52.264674 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-67f56e\"" Oct 30 12:58:52.277419 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:52.277389 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-67f56e-dockercfg-qb622\"" Oct 30 12:58:52.279678 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:52.279651 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78"] Oct 30 12:58:52.290708 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:52.290672 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5kl4\" (UniqueName: \"kubernetes.io/projected/c3d6937c-4924-4bb7-9449-6c4df4306904-kube-api-access-n5kl4\") pod \"payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78\" (UID: \"c3d6937c-4924-4bb7-9449-6c4df4306904\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78" Oct 30 12:58:52.290825 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:52.290725 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3d6937c-4924-4bb7-9449-6c4df4306904-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78\" (UID: \"c3d6937c-4924-4bb7-9449-6c4df4306904\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78" Oct 30 12:58:52.356132 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:52.356062 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt" event={"ID":"03309fcc-c2dd-4424-9deb-c34596cd1558","Type":"ContainerStarted","Data":"ff901eab67367b166da25cdddbf18beb67acbc1c54924922c824008bce2c248e"} Oct 30 12:58:52.356132 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:52.356111 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt" event={"ID":"03309fcc-c2dd-4424-9deb-c34596cd1558","Type":"ContainerStarted","Data":"7c887ec035061014e2ccf3ddbeb41d6bc5d9d72dac92703c5f2a957ffeee2bd2"} Oct 30 12:58:52.356359 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:52.356177 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt" Oct 30 12:58:52.357831 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:52.357803 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb" event={"ID":"440d993b-e62e-4284-955d-2e79adccec2e","Type":"ContainerStarted","Data":"f437a14b7e3a5be4eb14ad1f26a65ad40adbaa59dd34f6b1a5823245b7081ce2"} Oct 30 12:58:52.357930 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:52.357883 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb" Oct 30 12:58:52.392141 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:52.392112 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5kl4\" (UniqueName: \"kubernetes.io/projected/c3d6937c-4924-4bb7-9449-6c4df4306904-kube-api-access-n5kl4\") pod \"payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78\" (UID: \"c3d6937c-4924-4bb7-9449-6c4df4306904\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78" Oct 30 12:58:52.392259 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:52.392152 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3d6937c-4924-4bb7-9449-6c4df4306904-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78\" (UID: \"c3d6937c-4924-4bb7-9449-6c4df4306904\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78" Oct 30 12:58:52.393014 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:52.392996 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3d6937c-4924-4bb7-9449-6c4df4306904-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78\" (UID: \"c3d6937c-4924-4bb7-9449-6c4df4306904\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78" Oct 30 12:58:52.400495 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:52.400456 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt" podStartSLOduration=1.098331746 podStartE2EDuration="1.400446338s" totalImagesPullingTime="302.114592ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:52.398969758 +0000 UTC m=+831.002468208" watchObservedRunningTime="2026-10-30 12:58:52.400446338 +0000 UTC m=+831.003944784" Oct 30 12:58:52.416699 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:52.416677 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5kl4\" (UniqueName: \"kubernetes.io/projected/c3d6937c-4924-4bb7-9449-6c4df4306904-kube-api-access-n5kl4\") pod \"payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78\" (UID: \"c3d6937c-4924-4bb7-9449-6c4df4306904\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78" Oct 30 12:58:52.443499 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:52.443458 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb" podStartSLOduration=1.401187715 podStartE2EDuration="3.443446484s" totalImagesPullingTime="2.042258769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:52.441146694 +0000 UTC m=+831.044645139" watchObservedRunningTime="2026-10-30 12:58:52.443446484 +0000 UTC m=+831.046944929" Oct 30 12:58:52.563920 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:52.563879 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78" Oct 30 12:58:52.878826 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:52.878802 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78"] Oct 30 12:58:52.880508 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:58:52.880476 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3d6937c_4924_4bb7_9449_6c4df4306904.slice/crio-9ed968ef83be3bec0f5acab3d78021b94fcb33addc3b962bc7fbbc35ce1ba5ac WatchSource:0}: Error finding container 9ed968ef83be3bec0f5acab3d78021b94fcb33addc3b962bc7fbbc35ce1ba5ac: Status 404 returned error can't find the container with id 9ed968ef83be3bec0f5acab3d78021b94fcb33addc3b962bc7fbbc35ce1ba5ac Oct 30 12:58:53.366103 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:53.366065 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78" event={"ID":"c3d6937c-4924-4bb7-9449-6c4df4306904","Type":"ContainerStarted","Data":"8fe3e9ad466c7c28d914b2c28979c8cda7b4b7dacac69fc4070474b540ae0f0e"} Oct 30 12:58:53.366325 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:53.366125 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78" Oct 30 12:58:53.366325 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:53.366144 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78" event={"ID":"c3d6937c-4924-4bb7-9449-6c4df4306904","Type":"ContainerStarted","Data":"9ed968ef83be3bec0f5acab3d78021b94fcb33addc3b962bc7fbbc35ce1ba5ac"} Oct 30 12:58:53.413011 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:53.412967 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78" podStartSLOduration=1.412955122 podStartE2EDuration="1.412955122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:53.411798255 +0000 UTC m=+832.015296701" watchObservedRunningTime="2026-10-30 12:58:53.412955122 +0000 UTC m=+832.016453567" Oct 30 12:58:55.541358 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:58:55.541326 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 12:59:00.472228 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:00.472191 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 12:59:02.005068 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:02.005036 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx"] Oct 30 12:59:02.009876 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:02.009855 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx" Oct 30 12:59:02.047677 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:02.047642 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-67f56e-serving-cert\"" Oct 30 12:59:02.047677 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:02.047657 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:59:02.047901 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:02.047725 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 12:59:02.047901 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:02.047730 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:59:02.049298 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:02.049260 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nb7kb\"" Oct 30 12:59:02.050195 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:02.050171 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx"] Oct 30 12:59:02.082998 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:02.082972 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/996e2b88-ae9e-4e95-8a65-4180de0588b0-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx\" (UID: \"996e2b88-ae9e-4e95-8a65-4180de0588b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx" Oct 30 12:59:02.083108 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:02.083078 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/996e2b88-ae9e-4e95-8a65-4180de0588b0-maas-api-tls\") pod \"maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx\" (UID: \"996e2b88-ae9e-4e95-8a65-4180de0588b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx" Oct 30 12:59:02.083166 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:02.083145 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6c4d\" (UniqueName: \"kubernetes.io/projected/996e2b88-ae9e-4e95-8a65-4180de0588b0-kube-api-access-t6c4d\") pod \"maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx\" (UID: \"996e2b88-ae9e-4e95-8a65-4180de0588b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx" Oct 30 12:59:02.083262 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:02.083247 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/996e2b88-ae9e-4e95-8a65-4180de0588b0-metrics-certs\") pod \"maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx\" (UID: \"996e2b88-ae9e-4e95-8a65-4180de0588b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx" Oct 30 12:59:02.184688 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:02.184656 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/996e2b88-ae9e-4e95-8a65-4180de0588b0-maas-api-tls\") pod \"maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx\" (UID: \"996e2b88-ae9e-4e95-8a65-4180de0588b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx" Oct 30 12:59:02.184688 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:02.184693 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6c4d\" (UniqueName: \"kubernetes.io/projected/996e2b88-ae9e-4e95-8a65-4180de0588b0-kube-api-access-t6c4d\") pod \"maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx\" (UID: \"996e2b88-ae9e-4e95-8a65-4180de0588b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx" Oct 30 12:59:02.184872 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:02.184748 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/996e2b88-ae9e-4e95-8a65-4180de0588b0-metrics-certs\") pod \"maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx\" (UID: \"996e2b88-ae9e-4e95-8a65-4180de0588b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx" Oct 30 12:59:02.184872 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:02.184771 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/996e2b88-ae9e-4e95-8a65-4180de0588b0-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx\" (UID: \"996e2b88-ae9e-4e95-8a65-4180de0588b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx" Oct 30 12:59:02.184872 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:59:02.184814 4931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-67f56e-serving-cert: secret "maas-api-e2e-worker-w4-67f56e-serving-cert" not found Oct 30 12:59:02.185018 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:59:02.184902 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/996e2b88-ae9e-4e95-8a65-4180de0588b0-maas-api-tls podName:996e2b88-ae9e-4e95-8a65-4180de0588b0 nodeName:}" failed. No retries permitted until 2026-10-30 12:59:02.684884858 +0000 UTC m=+841.288383305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/996e2b88-ae9e-4e95-8a65-4180de0588b0-maas-api-tls") pod "maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx" (UID: "996e2b88-ae9e-4e95-8a65-4180de0588b0") : secret "maas-api-e2e-worker-w4-67f56e-serving-cert" not found Oct 30 12:59:02.185402 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:02.185375 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/996e2b88-ae9e-4e95-8a65-4180de0588b0-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx\" (UID: \"996e2b88-ae9e-4e95-8a65-4180de0588b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx" Oct 30 12:59:02.187201 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:02.187180 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/996e2b88-ae9e-4e95-8a65-4180de0588b0-metrics-certs\") pod \"maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx\" (UID: \"996e2b88-ae9e-4e95-8a65-4180de0588b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx" Oct 30 12:59:02.217437 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:02.217405 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6c4d\" (UniqueName: \"kubernetes.io/projected/996e2b88-ae9e-4e95-8a65-4180de0588b0-kube-api-access-t6c4d\") pod \"maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx\" (UID: \"996e2b88-ae9e-4e95-8a65-4180de0588b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx" Oct 30 12:59:02.689778 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:02.689742 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/996e2b88-ae9e-4e95-8a65-4180de0588b0-maas-api-tls\") pod \"maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx\" (UID: \"996e2b88-ae9e-4e95-8a65-4180de0588b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx" Oct 30 12:59:02.692112 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:02.692077 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/996e2b88-ae9e-4e95-8a65-4180de0588b0-maas-api-tls\") pod \"maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx\" (UID: \"996e2b88-ae9e-4e95-8a65-4180de0588b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx" Oct 30 12:59:02.922206 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:02.922165 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx" Oct 30 12:59:03.223087 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:03.223062 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx"] Oct 30 12:59:03.224893 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:59:03.224853 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod996e2b88_ae9e_4e95_8a65_4180de0588b0.slice/crio-9a7b63b825fe8511a7fa60b3cd9b61143bda3fdeb3a4b89319a3d09f6e5a5082 WatchSource:0}: Error finding container 9a7b63b825fe8511a7fa60b3cd9b61143bda3fdeb3a4b89319a3d09f6e5a5082: Status 404 returned error can't find the container with id 9a7b63b825fe8511a7fa60b3cd9b61143bda3fdeb3a4b89319a3d09f6e5a5082 Oct 30 12:59:03.369033 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:03.369005 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt" Oct 30 12:59:03.369219 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:03.369077 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb" Oct 30 12:59:03.420776 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:03.420732 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx" event={"ID":"996e2b88-ae9e-4e95-8a65-4180de0588b0","Type":"ContainerStarted","Data":"9a7b63b825fe8511a7fa60b3cd9b61143bda3fdeb3a4b89319a3d09f6e5a5082"} Oct 30 12:59:04.374988 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:04.374962 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78" Oct 30 12:59:06.160131 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:06.160094 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c"] Oct 30 12:59:06.164567 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:06.164548 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 12:59:06.186721 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:06.186686 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c90b9e\"/\"kube-root-ca.crt\"" Oct 30 12:59:06.187011 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:06.186991 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c90b9e\"/\"default-dockercfg-v2t7t\"" Oct 30 12:59:06.187011 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:06.187000 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c90b9e\"/\"openshift-service-ca.crt\"" Oct 30 12:59:06.199793 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:06.199771 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c90b9e\"/\"simulator-w0-c90b9e-kserve-self-signed-certs\"" Oct 30 12:59:06.202993 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:06.202968 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c"] Oct 30 12:59:06.221878 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:06.221852 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d345fdd5-407b-4162-a2cd-64a40068cb3c-model-cache\") pod \"simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c\" (UID: \"d345fdd5-407b-4162-a2cd-64a40068cb3c\") " pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 12:59:06.222008 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:06.221884 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d345fdd5-407b-4162-a2cd-64a40068cb3c-tls-certs\") pod \"simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c\" (UID: \"d345fdd5-407b-4162-a2cd-64a40068cb3c\") " pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 12:59:06.222008 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:06.221914 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq79j\" (UniqueName: \"kubernetes.io/projected/d345fdd5-407b-4162-a2cd-64a40068cb3c-kube-api-access-fq79j\") pod \"simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c\" (UID: \"d345fdd5-407b-4162-a2cd-64a40068cb3c\") " pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 12:59:06.222008 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:06.221977 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d345fdd5-407b-4162-a2cd-64a40068cb3c-tmp-dir\") pod \"simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c\" (UID: \"d345fdd5-407b-4162-a2cd-64a40068cb3c\") " pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 12:59:06.222174 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:06.222092 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d345fdd5-407b-4162-a2cd-64a40068cb3c-home\") pod \"simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c\" (UID: \"d345fdd5-407b-4162-a2cd-64a40068cb3c\") " pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 12:59:06.222174 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:06.222141 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d345fdd5-407b-4162-a2cd-64a40068cb3c-dshm\") pod \"simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c\" (UID: \"d345fdd5-407b-4162-a2cd-64a40068cb3c\") " pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 12:59:06.323175 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:06.323142 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d345fdd5-407b-4162-a2cd-64a40068cb3c-dshm\") pod \"simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c\" (UID: \"d345fdd5-407b-4162-a2cd-64a40068cb3c\") " pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 12:59:06.323399 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:06.323235 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d345fdd5-407b-4162-a2cd-64a40068cb3c-model-cache\") pod \"simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c\" (UID: \"d345fdd5-407b-4162-a2cd-64a40068cb3c\") " pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 12:59:06.323399 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:06.323256 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d345fdd5-407b-4162-a2cd-64a40068cb3c-tls-certs\") pod \"simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c\" (UID: \"d345fdd5-407b-4162-a2cd-64a40068cb3c\") " pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 12:59:06.323399 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:06.323305 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq79j\" (UniqueName: \"kubernetes.io/projected/d345fdd5-407b-4162-a2cd-64a40068cb3c-kube-api-access-fq79j\") pod \"simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c\" (UID: \"d345fdd5-407b-4162-a2cd-64a40068cb3c\") " pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 12:59:06.323399 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:06.323346 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d345fdd5-407b-4162-a2cd-64a40068cb3c-tmp-dir\") pod \"simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c\" (UID: \"d345fdd5-407b-4162-a2cd-64a40068cb3c\") " pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 12:59:06.323399 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:06.323380 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d345fdd5-407b-4162-a2cd-64a40068cb3c-home\") pod \"simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c\" (UID: \"d345fdd5-407b-4162-a2cd-64a40068cb3c\") " pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 12:59:06.323731 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:06.323704 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d345fdd5-407b-4162-a2cd-64a40068cb3c-model-cache\") pod \"simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c\" (UID: \"d345fdd5-407b-4162-a2cd-64a40068cb3c\") " pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 12:59:06.323731 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:06.323724 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d345fdd5-407b-4162-a2cd-64a40068cb3c-tmp-dir\") pod \"simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c\" (UID: \"d345fdd5-407b-4162-a2cd-64a40068cb3c\") " pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 12:59:06.323846 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:06.323783 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d345fdd5-407b-4162-a2cd-64a40068cb3c-home\") pod \"simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c\" (UID: \"d345fdd5-407b-4162-a2cd-64a40068cb3c\") " pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 12:59:06.325599 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:06.325568 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d345fdd5-407b-4162-a2cd-64a40068cb3c-dshm\") pod \"simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c\" (UID: \"d345fdd5-407b-4162-a2cd-64a40068cb3c\") " pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 12:59:06.325887 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:06.325865 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d345fdd5-407b-4162-a2cd-64a40068cb3c-tls-certs\") pod \"simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c\" (UID: \"d345fdd5-407b-4162-a2cd-64a40068cb3c\") " pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 12:59:06.362553 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:06.362521 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq79j\" (UniqueName: \"kubernetes.io/projected/d345fdd5-407b-4162-a2cd-64a40068cb3c-kube-api-access-fq79j\") pod \"simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c\" (UID: \"d345fdd5-407b-4162-a2cd-64a40068cb3c\") " pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 12:59:06.444069 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:06.444037 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx" event={"ID":"996e2b88-ae9e-4e95-8a65-4180de0588b0","Type":"ContainerStarted","Data":"f77d9c53b26327d3144a16df810b5d0726b031c2da39f49f5bf081f789407650"} Oct 30 12:59:06.444317 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:06.444293 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx" Oct 30 12:59:06.475995 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:06.475960 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 12:59:06.498546 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:06.498495 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx" podStartSLOduration=3.176320444 podStartE2EDuration="5.498478484s" totalImagesPullingTime="2.32215804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:06.496429105 +0000 UTC m=+845.099927569" watchObservedRunningTime="2026-10-30 12:59:06.498478484 +0000 UTC m=+845.101976930" Oct 30 12:59:06.787220 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:06.787188 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c"] Oct 30 12:59:06.788681 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:59:06.788654 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd345fdd5_407b_4162_a2cd_64a40068cb3c.slice/crio-32ce3755ba62b725d915ac447d981316c121527a0d146509aa338c18ce849bc5 WatchSource:0}: Error finding container 32ce3755ba62b725d915ac447d981316c121527a0d146509aa338c18ce849bc5: Status 404 returned error can't find the container with id 32ce3755ba62b725d915ac447d981316c121527a0d146509aa338c18ce849bc5 Oct 30 12:59:07.366971 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:59:07.366928 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 12:59:07.453097 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:07.453055 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" event={"ID":"d345fdd5-407b-4162-a2cd-64a40068cb3c","Type":"ContainerStarted","Data":"32ce3755ba62b725d915ac447d981316c121527a0d146509aa338c18ce849bc5"} Oct 30 12:59:08.466086 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:08.466044 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" event={"ID":"d345fdd5-407b-4162-a2cd-64a40068cb3c","Type":"ContainerStarted","Data":"a9dc518c33e4978fb35eaf3bc23a73e29be95c7b9e8d6b23b339a665eb776fcf"} Oct 30 12:59:08.476105 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:08.476082 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 12:59:08.511041 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:08.510992 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" podStartSLOduration=1.333296951 podStartE2EDuration="2.510979966s" totalImagesPullingTime="1.177683015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:08.510028327 +0000 UTC m=+847.113526772" watchObservedRunningTime="2026-10-30 12:59:08.510979966 +0000 UTC m=+847.114478410" Oct 30 12:59:12.459303 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:12.459258 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx" Oct 30 12:59:13.620220 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:13.620189 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 12:59:16.477129 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:16.477095 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 12:59:16.489559 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:16.489532 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 12:59:16.518950 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:16.518927 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 12:59:19.696097 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:19.696051 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj"] Oct 30 12:59:19.700543 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:19.700512 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 12:59:19.717794 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:19.717762 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj"] Oct 30 12:59:19.729287 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:19.729243 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-dd90cb\"/\"kube-root-ca.crt\"" Oct 30 12:59:19.730155 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:19.730133 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-dd90cb\"/\"openshift-service-ca.crt\"" Oct 30 12:59:19.730261 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:19.730159 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-dd90cb\"/\"simulator-w6-dd90cb-kserve-self-signed-certs\"" Oct 30 12:59:19.730261 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:19.730161 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-dd90cb\"/\"default-dockercfg-69qp9\"" Oct 30 12:59:19.852461 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:19.852414 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jjd2\" (UniqueName: \"kubernetes.io/projected/606b252b-881c-4d40-8ba9-fc4ed937a995-kube-api-access-2jjd2\") pod \"simulator-w6-dd90cb-kserve-77cd5c969-s8xqj\" (UID: \"606b252b-881c-4d40-8ba9-fc4ed937a995\") " pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 12:59:19.852665 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:19.852559 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/606b252b-881c-4d40-8ba9-fc4ed937a995-model-cache\") pod \"simulator-w6-dd90cb-kserve-77cd5c969-s8xqj\" (UID: \"606b252b-881c-4d40-8ba9-fc4ed937a995\") " pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 12:59:19.852665 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:19.852583 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/606b252b-881c-4d40-8ba9-fc4ed937a995-tls-certs\") pod \"simulator-w6-dd90cb-kserve-77cd5c969-s8xqj\" (UID: \"606b252b-881c-4d40-8ba9-fc4ed937a995\") " pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 12:59:19.852782 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:19.852711 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/606b252b-881c-4d40-8ba9-fc4ed937a995-home\") pod \"simulator-w6-dd90cb-kserve-77cd5c969-s8xqj\" (UID: \"606b252b-881c-4d40-8ba9-fc4ed937a995\") " pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 12:59:19.852838 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:19.852798 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/606b252b-881c-4d40-8ba9-fc4ed937a995-dshm\") pod \"simulator-w6-dd90cb-kserve-77cd5c969-s8xqj\" (UID: \"606b252b-881c-4d40-8ba9-fc4ed937a995\") " pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 12:59:19.852899 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:19.852839 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/606b252b-881c-4d40-8ba9-fc4ed937a995-tmp-dir\") pod \"simulator-w6-dd90cb-kserve-77cd5c969-s8xqj\" (UID: \"606b252b-881c-4d40-8ba9-fc4ed937a995\") " pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 12:59:19.954065 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:19.953974 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/606b252b-881c-4d40-8ba9-fc4ed937a995-model-cache\") pod \"simulator-w6-dd90cb-kserve-77cd5c969-s8xqj\" (UID: \"606b252b-881c-4d40-8ba9-fc4ed937a995\") " pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 12:59:19.954065 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:19.954014 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/606b252b-881c-4d40-8ba9-fc4ed937a995-tls-certs\") pod \"simulator-w6-dd90cb-kserve-77cd5c969-s8xqj\" (UID: \"606b252b-881c-4d40-8ba9-fc4ed937a995\") " pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 12:59:19.954065 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:19.954051 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/606b252b-881c-4d40-8ba9-fc4ed937a995-home\") pod \"simulator-w6-dd90cb-kserve-77cd5c969-s8xqj\" (UID: \"606b252b-881c-4d40-8ba9-fc4ed937a995\") " pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 12:59:19.954314 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:19.954071 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/606b252b-881c-4d40-8ba9-fc4ed937a995-dshm\") pod \"simulator-w6-dd90cb-kserve-77cd5c969-s8xqj\" (UID: \"606b252b-881c-4d40-8ba9-fc4ed937a995\") " pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 12:59:19.954314 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:19.954108 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/606b252b-881c-4d40-8ba9-fc4ed937a995-tmp-dir\") pod \"simulator-w6-dd90cb-kserve-77cd5c969-s8xqj\" (UID: \"606b252b-881c-4d40-8ba9-fc4ed937a995\") " pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 12:59:19.954314 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:19.954141 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jjd2\" (UniqueName: \"kubernetes.io/projected/606b252b-881c-4d40-8ba9-fc4ed937a995-kube-api-access-2jjd2\") pod \"simulator-w6-dd90cb-kserve-77cd5c969-s8xqj\" (UID: \"606b252b-881c-4d40-8ba9-fc4ed937a995\") " pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 12:59:19.954867 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:19.954848 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/606b252b-881c-4d40-8ba9-fc4ed937a995-model-cache\") pod \"simulator-w6-dd90cb-kserve-77cd5c969-s8xqj\" (UID: \"606b252b-881c-4d40-8ba9-fc4ed937a995\") " pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 12:59:19.955470 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:19.955444 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/606b252b-881c-4d40-8ba9-fc4ed937a995-home\") pod \"simulator-w6-dd90cb-kserve-77cd5c969-s8xqj\" (UID: \"606b252b-881c-4d40-8ba9-fc4ed937a995\") " pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 12:59:19.955655 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:19.955509 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/606b252b-881c-4d40-8ba9-fc4ed937a995-tmp-dir\") pod \"simulator-w6-dd90cb-kserve-77cd5c969-s8xqj\" (UID: \"606b252b-881c-4d40-8ba9-fc4ed937a995\") " pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 12:59:19.957905 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:19.957883 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/606b252b-881c-4d40-8ba9-fc4ed937a995-dshm\") pod \"simulator-w6-dd90cb-kserve-77cd5c969-s8xqj\" (UID: \"606b252b-881c-4d40-8ba9-fc4ed937a995\") " pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 12:59:19.958051 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:19.958020 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/606b252b-881c-4d40-8ba9-fc4ed937a995-tls-certs\") pod \"simulator-w6-dd90cb-kserve-77cd5c969-s8xqj\" (UID: \"606b252b-881c-4d40-8ba9-fc4ed937a995\") " pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 12:59:19.978523 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:19.978496 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jjd2\" (UniqueName: \"kubernetes.io/projected/606b252b-881c-4d40-8ba9-fc4ed937a995-kube-api-access-2jjd2\") pod \"simulator-w6-dd90cb-kserve-77cd5c969-s8xqj\" (UID: \"606b252b-881c-4d40-8ba9-fc4ed937a995\") " pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 12:59:20.013431 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:20.013410 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 12:59:20.319111 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:20.319085 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj"] Oct 30 12:59:20.320578 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:59:20.320550 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod606b252b_881c_4d40_8ba9_fc4ed937a995.slice/crio-cf907cf68a3b69579ea545e3f069fc6c02d3c482084a60a115bf7363f51c0198 WatchSource:0}: Error finding container cf907cf68a3b69579ea545e3f069fc6c02d3c482084a60a115bf7363f51c0198: Status 404 returned error can't find the container with id cf907cf68a3b69579ea545e3f069fc6c02d3c482084a60a115bf7363f51c0198 Oct 30 12:59:20.536089 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:20.536053 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" event={"ID":"606b252b-881c-4d40-8ba9-fc4ed937a995","Type":"ContainerStarted","Data":"7d68128c0f65505a22952b820ae87b9c0472ab6b2a1a75b8cdac2878b6eae845"} Oct 30 12:59:20.536257 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:20.536103 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" event={"ID":"606b252b-881c-4d40-8ba9-fc4ed937a995","Type":"ContainerStarted","Data":"cf907cf68a3b69579ea545e3f069fc6c02d3c482084a60a115bf7363f51c0198"} Oct 30 12:59:20.582926 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:20.582838 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" podStartSLOduration=1.582825883 podStartE2EDuration="1.582825883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:20.580430593 +0000 UTC m=+859.183929039" watchObservedRunningTime="2026-10-30 12:59:20.582825883 +0000 UTC m=+859.186324329" Oct 30 12:59:21.013944 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:21.013910 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 12:59:23.029845 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:23.029810 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 12:59:30.014064 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:30.014031 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 12:59:30.026567 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:30.026540 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 12:59:30.603308 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:30.603281 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 12:59:35.013073 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:35.013029 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 12:59:35.739140 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:35.739108 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf"] Oct 30 12:59:35.747387 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:35.747367 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 12:59:35.778102 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:35.778072 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c90b9e\"/\"premium-w0-c90b9e-kserve-self-signed-certs\"" Oct 30 12:59:35.794829 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:35.794799 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf"] Oct 30 12:59:35.810829 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:35.810804 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f28bn\" (UniqueName: \"kubernetes.io/projected/0c5a9f00-6d5b-4860-a210-2fec24afb322-kube-api-access-f28bn\") pod \"premium-w0-c90b9e-kserve-5cdc445666-4n5lf\" (UID: \"0c5a9f00-6d5b-4860-a210-2fec24afb322\") " pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 12:59:35.810966 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:35.810859 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c5a9f00-6d5b-4860-a210-2fec24afb322-home\") pod \"premium-w0-c90b9e-kserve-5cdc445666-4n5lf\" (UID: \"0c5a9f00-6d5b-4860-a210-2fec24afb322\") " pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 12:59:35.810966 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:35.810879 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c5a9f00-6d5b-4860-a210-2fec24afb322-tls-certs\") pod \"premium-w0-c90b9e-kserve-5cdc445666-4n5lf\" (UID: \"0c5a9f00-6d5b-4860-a210-2fec24afb322\") " pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 12:59:35.811087 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:35.811023 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c5a9f00-6d5b-4860-a210-2fec24afb322-model-cache\") pod \"premium-w0-c90b9e-kserve-5cdc445666-4n5lf\" (UID: \"0c5a9f00-6d5b-4860-a210-2fec24afb322\") " pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 12:59:35.811087 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:35.811072 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c5a9f00-6d5b-4860-a210-2fec24afb322-tmp-dir\") pod \"premium-w0-c90b9e-kserve-5cdc445666-4n5lf\" (UID: \"0c5a9f00-6d5b-4860-a210-2fec24afb322\") " pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 12:59:35.811224 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:35.811198 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c5a9f00-6d5b-4860-a210-2fec24afb322-dshm\") pod \"premium-w0-c90b9e-kserve-5cdc445666-4n5lf\" (UID: \"0c5a9f00-6d5b-4860-a210-2fec24afb322\") " pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 12:59:35.912476 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:35.912437 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c5a9f00-6d5b-4860-a210-2fec24afb322-model-cache\") pod \"premium-w0-c90b9e-kserve-5cdc445666-4n5lf\" (UID: \"0c5a9f00-6d5b-4860-a210-2fec24afb322\") " pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 12:59:35.912476 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:35.912481 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c5a9f00-6d5b-4860-a210-2fec24afb322-tmp-dir\") pod \"premium-w0-c90b9e-kserve-5cdc445666-4n5lf\" (UID: \"0c5a9f00-6d5b-4860-a210-2fec24afb322\") " pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 12:59:35.912761 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:35.912514 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c5a9f00-6d5b-4860-a210-2fec24afb322-dshm\") pod \"premium-w0-c90b9e-kserve-5cdc445666-4n5lf\" (UID: \"0c5a9f00-6d5b-4860-a210-2fec24afb322\") " pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 12:59:35.912761 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:35.912554 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f28bn\" (UniqueName: \"kubernetes.io/projected/0c5a9f00-6d5b-4860-a210-2fec24afb322-kube-api-access-f28bn\") pod \"premium-w0-c90b9e-kserve-5cdc445666-4n5lf\" (UID: \"0c5a9f00-6d5b-4860-a210-2fec24afb322\") " pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 12:59:35.912761 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:35.912599 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c5a9f00-6d5b-4860-a210-2fec24afb322-home\") pod \"premium-w0-c90b9e-kserve-5cdc445666-4n5lf\" (UID: \"0c5a9f00-6d5b-4860-a210-2fec24afb322\") " pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 12:59:35.912761 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:35.912621 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c5a9f00-6d5b-4860-a210-2fec24afb322-tls-certs\") pod \"premium-w0-c90b9e-kserve-5cdc445666-4n5lf\" (UID: \"0c5a9f00-6d5b-4860-a210-2fec24afb322\") " pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 12:59:35.912967 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:35.912924 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c5a9f00-6d5b-4860-a210-2fec24afb322-model-cache\") pod \"premium-w0-c90b9e-kserve-5cdc445666-4n5lf\" (UID: \"0c5a9f00-6d5b-4860-a210-2fec24afb322\") " pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 12:59:35.913029 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:35.912966 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c5a9f00-6d5b-4860-a210-2fec24afb322-tmp-dir\") pod \"premium-w0-c90b9e-kserve-5cdc445666-4n5lf\" (UID: \"0c5a9f00-6d5b-4860-a210-2fec24afb322\") " pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 12:59:35.913029 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:35.913018 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c5a9f00-6d5b-4860-a210-2fec24afb322-home\") pod \"premium-w0-c90b9e-kserve-5cdc445666-4n5lf\" (UID: \"0c5a9f00-6d5b-4860-a210-2fec24afb322\") " pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 12:59:35.914847 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:35.914819 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c5a9f00-6d5b-4860-a210-2fec24afb322-dshm\") pod \"premium-w0-c90b9e-kserve-5cdc445666-4n5lf\" (UID: \"0c5a9f00-6d5b-4860-a210-2fec24afb322\") " pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 12:59:35.915159 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:35.915131 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c5a9f00-6d5b-4860-a210-2fec24afb322-tls-certs\") pod \"premium-w0-c90b9e-kserve-5cdc445666-4n5lf\" (UID: \"0c5a9f00-6d5b-4860-a210-2fec24afb322\") " pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 12:59:35.948547 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:35.948524 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f28bn\" (UniqueName: \"kubernetes.io/projected/0c5a9f00-6d5b-4860-a210-2fec24afb322-kube-api-access-f28bn\") pod \"premium-w0-c90b9e-kserve-5cdc445666-4n5lf\" (UID: \"0c5a9f00-6d5b-4860-a210-2fec24afb322\") " pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 12:59:36.061404 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:36.061309 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 12:59:36.367009 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:36.366981 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf"] Oct 30 12:59:36.368814 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:59:36.368790 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c5a9f00_6d5b_4860_a210_2fec24afb322.slice/crio-a33a7de8fe933f94afc351ca53c207dfa49a6588d862ac0f2a66484e23aebd66 WatchSource:0}: Error finding container a33a7de8fe933f94afc351ca53c207dfa49a6588d862ac0f2a66484e23aebd66: Status 404 returned error can't find the container with id a33a7de8fe933f94afc351ca53c207dfa49a6588d862ac0f2a66484e23aebd66 Oct 30 12:59:36.625781 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:36.625689 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" event={"ID":"0c5a9f00-6d5b-4860-a210-2fec24afb322","Type":"ContainerStarted","Data":"42316409e0da023240a8e21ef23c9ddc169a8cc3c18e57e2d1cfbb26d722b4bc"} Oct 30 12:59:36.625781 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:36.625741 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" event={"ID":"0c5a9f00-6d5b-4860-a210-2fec24afb322","Type":"ContainerStarted","Data":"a33a7de8fe933f94afc351ca53c207dfa49a6588d862ac0f2a66484e23aebd66"} Oct 30 12:59:36.671099 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:36.671037 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" podStartSLOduration=1.671024332 podStartE2EDuration="1.671024332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:36.669831648 +0000 UTC m=+875.273330107" watchObservedRunningTime="2026-10-30 12:59:36.671024332 +0000 UTC m=+875.274522777" Oct 30 12:59:37.062391 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:37.062354 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 12:59:37.352804 ip-10-0-142-194 kubenswrapper[4931]: E1030 12:59:37.352714 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 12:59:45.302288 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:45.302228 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht"] Oct 30 12:59:45.306120 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:45.306100 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" Oct 30 12:59:45.322914 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:45.322893 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-67f56e\"/\"kube-root-ca.crt\"" Oct 30 12:59:45.323072 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:45.323032 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-67f56e\"/\"openshift-service-ca.crt\"" Oct 30 12:59:45.336424 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:45.336399 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-67f56e\"/\"default-dockercfg-7ldk6\"" Oct 30 12:59:45.336585 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:45.336535 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-67f56e\"/\"simulator-w4-67f56e-kserve-self-signed-certs\"" Oct 30 12:59:45.339035 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:45.339015 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht"] Oct 30 12:59:45.409099 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:45.409059 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-tls-certs\") pod \"simulator-w4-67f56e-kserve-69f8c59df8-tt5ht\" (UID: \"b1e86eea-4824-4c50-8ad3-2a15c3365c3d\") " pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" Oct 30 12:59:45.409099 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:45.409098 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-dshm\") pod \"simulator-w4-67f56e-kserve-69f8c59df8-tt5ht\" (UID: \"b1e86eea-4824-4c50-8ad3-2a15c3365c3d\") " pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" Oct 30 12:59:45.409387 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:45.409120 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-model-cache\") pod \"simulator-w4-67f56e-kserve-69f8c59df8-tt5ht\" (UID: \"b1e86eea-4824-4c50-8ad3-2a15c3365c3d\") " pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" Oct 30 12:59:45.409387 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:45.409225 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8glx9\" (UniqueName: \"kubernetes.io/projected/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-kube-api-access-8glx9\") pod \"simulator-w4-67f56e-kserve-69f8c59df8-tt5ht\" (UID: \"b1e86eea-4824-4c50-8ad3-2a15c3365c3d\") " pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" Oct 30 12:59:45.409387 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:45.409327 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-home\") pod \"simulator-w4-67f56e-kserve-69f8c59df8-tt5ht\" (UID: \"b1e86eea-4824-4c50-8ad3-2a15c3365c3d\") " pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" Oct 30 12:59:45.409387 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:45.409349 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-tmp-dir\") pod \"simulator-w4-67f56e-kserve-69f8c59df8-tt5ht\" (UID: \"b1e86eea-4824-4c50-8ad3-2a15c3365c3d\") " pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" Oct 30 12:59:45.510578 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:45.510544 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-tls-certs\") pod \"simulator-w4-67f56e-kserve-69f8c59df8-tt5ht\" (UID: \"b1e86eea-4824-4c50-8ad3-2a15c3365c3d\") " pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" Oct 30 12:59:45.510578 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:45.510581 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-dshm\") pod \"simulator-w4-67f56e-kserve-69f8c59df8-tt5ht\" (UID: \"b1e86eea-4824-4c50-8ad3-2a15c3365c3d\") " pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" Oct 30 12:59:45.510912 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:45.510600 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-model-cache\") pod \"simulator-w4-67f56e-kserve-69f8c59df8-tt5ht\" (UID: \"b1e86eea-4824-4c50-8ad3-2a15c3365c3d\") " pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" Oct 30 12:59:45.510912 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:45.510629 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8glx9\" (UniqueName: \"kubernetes.io/projected/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-kube-api-access-8glx9\") pod \"simulator-w4-67f56e-kserve-69f8c59df8-tt5ht\" (UID: \"b1e86eea-4824-4c50-8ad3-2a15c3365c3d\") " pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" Oct 30 12:59:45.510912 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:45.510714 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-home\") pod \"simulator-w4-67f56e-kserve-69f8c59df8-tt5ht\" (UID: \"b1e86eea-4824-4c50-8ad3-2a15c3365c3d\") " pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" Oct 30 12:59:45.510912 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:45.510737 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-tmp-dir\") pod \"simulator-w4-67f56e-kserve-69f8c59df8-tt5ht\" (UID: \"b1e86eea-4824-4c50-8ad3-2a15c3365c3d\") " pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" Oct 30 12:59:45.511107 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:45.511034 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-model-cache\") pod \"simulator-w4-67f56e-kserve-69f8c59df8-tt5ht\" (UID: \"b1e86eea-4824-4c50-8ad3-2a15c3365c3d\") " pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" Oct 30 12:59:45.511107 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:45.511068 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-tmp-dir\") pod \"simulator-w4-67f56e-kserve-69f8c59df8-tt5ht\" (UID: \"b1e86eea-4824-4c50-8ad3-2a15c3365c3d\") " pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" Oct 30 12:59:45.511379 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:45.511285 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-home\") pod \"simulator-w4-67f56e-kserve-69f8c59df8-tt5ht\" (UID: \"b1e86eea-4824-4c50-8ad3-2a15c3365c3d\") " pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" Oct 30 12:59:45.513440 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:45.513413 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-dshm\") pod \"simulator-w4-67f56e-kserve-69f8c59df8-tt5ht\" (UID: \"b1e86eea-4824-4c50-8ad3-2a15c3365c3d\") " pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" Oct 30 12:59:45.513599 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:45.513449 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-tls-certs\") pod \"simulator-w4-67f56e-kserve-69f8c59df8-tt5ht\" (UID: \"b1e86eea-4824-4c50-8ad3-2a15c3365c3d\") " pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" Oct 30 12:59:45.536785 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:45.536751 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8glx9\" (UniqueName: \"kubernetes.io/projected/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-kube-api-access-8glx9\") pod \"simulator-w4-67f56e-kserve-69f8c59df8-tt5ht\" (UID: \"b1e86eea-4824-4c50-8ad3-2a15c3365c3d\") " pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" Oct 30 12:59:45.619891 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:45.619806 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" Oct 30 12:59:45.926259 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:45.926223 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht"] Oct 30 12:59:45.926428 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:59:45.926293 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1e86eea_4824_4c50_8ad3_2a15c3365c3d.slice/crio-f3e6c8e59a8d06b63e09cca8e3a5e72a7b772e8c6edc51e5c1f21adaaa46213a WatchSource:0}: Error finding container f3e6c8e59a8d06b63e09cca8e3a5e72a7b772e8c6edc51e5c1f21adaaa46213a: Status 404 returned error can't find the container with id f3e6c8e59a8d06b63e09cca8e3a5e72a7b772e8c6edc51e5c1f21adaaa46213a Oct 30 12:59:46.061953 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:46.061914 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 12:59:46.074572 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:46.074546 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 12:59:46.697207 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:46.697170 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" event={"ID":"b1e86eea-4824-4c50-8ad3-2a15c3365c3d","Type":"ContainerStarted","Data":"9c10004e68dad22efe8c38e4ffcda9a991b30f1d514b10df9e39fa445e51a593"} Oct 30 12:59:46.697671 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:46.697218 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" event={"ID":"b1e86eea-4824-4c50-8ad3-2a15c3365c3d","Type":"ContainerStarted","Data":"f3e6c8e59a8d06b63e09cca8e3a5e72a7b772e8c6edc51e5c1f21adaaa46213a"} Oct 30 12:59:46.708292 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:46.708246 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 12:59:46.742876 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:46.742827 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" podStartSLOduration=1.742815963 podStartE2EDuration="1.742815963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:46.740932207 +0000 UTC m=+885.344430656" watchObservedRunningTime="2026-10-30 12:59:46.742815963 +0000 UTC m=+885.346314408" Oct 30 12:59:47.620577 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:47.620544 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" Oct 30 12:59:49.494197 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:49.494158 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 12:59:49.875051 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:49.874973 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84"] Oct 30 12:59:49.878931 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:49.878909 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 12:59:49.897419 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:49.897400 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-8b46c9\"/\"simulator-w1-8b46c9-kserve-self-signed-certs\"" Oct 30 12:59:49.910936 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:49.910891 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-8b46c9\"/\"default-dockercfg-kqjkl\"" Oct 30 12:59:49.913187 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:49.913164 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-8b46c9\"/\"openshift-service-ca.crt\"" Oct 30 12:59:49.921361 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:49.913635 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-8b46c9\"/\"kube-root-ca.crt\"" Oct 30 12:59:49.921597 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:49.921570 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84"] Oct 30 12:59:49.958779 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:49.958752 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64468005-61ea-4f2a-b72c-9c30c6a4d686-tmp-dir\") pod \"simulator-w1-8b46c9-kserve-7d57d6f588-ppb84\" (UID: \"64468005-61ea-4f2a-b72c-9c30c6a4d686\") " pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 12:59:49.958779 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:49.958783 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64468005-61ea-4f2a-b72c-9c30c6a4d686-tls-certs\") pod \"simulator-w1-8b46c9-kserve-7d57d6f588-ppb84\" (UID: \"64468005-61ea-4f2a-b72c-9c30c6a4d686\") " pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 12:59:49.958973 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:49.958818 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64468005-61ea-4f2a-b72c-9c30c6a4d686-dshm\") pod \"simulator-w1-8b46c9-kserve-7d57d6f588-ppb84\" (UID: \"64468005-61ea-4f2a-b72c-9c30c6a4d686\") " pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 12:59:49.958973 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:49.958891 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64468005-61ea-4f2a-b72c-9c30c6a4d686-home\") pod \"simulator-w1-8b46c9-kserve-7d57d6f588-ppb84\" (UID: \"64468005-61ea-4f2a-b72c-9c30c6a4d686\") " pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 12:59:49.958973 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:49.958963 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pmp9\" (UniqueName: \"kubernetes.io/projected/64468005-61ea-4f2a-b72c-9c30c6a4d686-kube-api-access-7pmp9\") pod \"simulator-w1-8b46c9-kserve-7d57d6f588-ppb84\" (UID: \"64468005-61ea-4f2a-b72c-9c30c6a4d686\") " pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 12:59:49.959069 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:49.959040 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64468005-61ea-4f2a-b72c-9c30c6a4d686-model-cache\") pod \"simulator-w1-8b46c9-kserve-7d57d6f588-ppb84\" (UID: \"64468005-61ea-4f2a-b72c-9c30c6a4d686\") " pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 12:59:50.060115 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:50.060083 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64468005-61ea-4f2a-b72c-9c30c6a4d686-dshm\") pod \"simulator-w1-8b46c9-kserve-7d57d6f588-ppb84\" (UID: \"64468005-61ea-4f2a-b72c-9c30c6a4d686\") " pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 12:59:50.060115 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:50.060117 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64468005-61ea-4f2a-b72c-9c30c6a4d686-home\") pod \"simulator-w1-8b46c9-kserve-7d57d6f588-ppb84\" (UID: \"64468005-61ea-4f2a-b72c-9c30c6a4d686\") " pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 12:59:50.060371 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:50.060151 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pmp9\" (UniqueName: \"kubernetes.io/projected/64468005-61ea-4f2a-b72c-9c30c6a4d686-kube-api-access-7pmp9\") pod \"simulator-w1-8b46c9-kserve-7d57d6f588-ppb84\" (UID: \"64468005-61ea-4f2a-b72c-9c30c6a4d686\") " pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 12:59:50.060371 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:50.060200 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64468005-61ea-4f2a-b72c-9c30c6a4d686-model-cache\") pod \"simulator-w1-8b46c9-kserve-7d57d6f588-ppb84\" (UID: \"64468005-61ea-4f2a-b72c-9c30c6a4d686\") " pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 12:59:50.060371 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:50.060237 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64468005-61ea-4f2a-b72c-9c30c6a4d686-tmp-dir\") pod \"simulator-w1-8b46c9-kserve-7d57d6f588-ppb84\" (UID: \"64468005-61ea-4f2a-b72c-9c30c6a4d686\") " pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 12:59:50.060371 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:50.060257 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64468005-61ea-4f2a-b72c-9c30c6a4d686-tls-certs\") pod \"simulator-w1-8b46c9-kserve-7d57d6f588-ppb84\" (UID: \"64468005-61ea-4f2a-b72c-9c30c6a4d686\") " pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 12:59:50.060744 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:50.060718 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64468005-61ea-4f2a-b72c-9c30c6a4d686-home\") pod \"simulator-w1-8b46c9-kserve-7d57d6f588-ppb84\" (UID: \"64468005-61ea-4f2a-b72c-9c30c6a4d686\") " pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 12:59:50.060846 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:50.060767 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64468005-61ea-4f2a-b72c-9c30c6a4d686-model-cache\") pod \"simulator-w1-8b46c9-kserve-7d57d6f588-ppb84\" (UID: \"64468005-61ea-4f2a-b72c-9c30c6a4d686\") " pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 12:59:50.060846 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:50.060816 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64468005-61ea-4f2a-b72c-9c30c6a4d686-tmp-dir\") pod \"simulator-w1-8b46c9-kserve-7d57d6f588-ppb84\" (UID: \"64468005-61ea-4f2a-b72c-9c30c6a4d686\") " pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 12:59:50.062733 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:50.062710 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64468005-61ea-4f2a-b72c-9c30c6a4d686-dshm\") pod \"simulator-w1-8b46c9-kserve-7d57d6f588-ppb84\" (UID: \"64468005-61ea-4f2a-b72c-9c30c6a4d686\") " pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 12:59:50.063384 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:50.063364 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64468005-61ea-4f2a-b72c-9c30c6a4d686-tls-certs\") pod \"simulator-w1-8b46c9-kserve-7d57d6f588-ppb84\" (UID: \"64468005-61ea-4f2a-b72c-9c30c6a4d686\") " pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 12:59:50.089310 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:50.089283 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pmp9\" (UniqueName: \"kubernetes.io/projected/64468005-61ea-4f2a-b72c-9c30c6a4d686-kube-api-access-7pmp9\") pod \"simulator-w1-8b46c9-kserve-7d57d6f588-ppb84\" (UID: \"64468005-61ea-4f2a-b72c-9c30c6a4d686\") " pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 12:59:50.191841 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:50.191757 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 12:59:50.520206 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:50.520177 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84"] Oct 30 12:59:50.520707 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:59:50.520678 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64468005_61ea_4f2a_b72c_9c30c6a4d686.slice/crio-782a2d893db6bb620cd47450c710e2e31bf94afb9d5989a6a97f1f631396990d WatchSource:0}: Error finding container 782a2d893db6bb620cd47450c710e2e31bf94afb9d5989a6a97f1f631396990d: Status 404 returned error can't find the container with id 782a2d893db6bb620cd47450c710e2e31bf94afb9d5989a6a97f1f631396990d Oct 30 12:59:50.721326 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:50.721290 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" event={"ID":"64468005-61ea-4f2a-b72c-9c30c6a4d686","Type":"ContainerStarted","Data":"b444cfe4e01b7d9d26ec02fcef11e83b5f216fc92d70f77157d1742768bc21fd"} Oct 30 12:59:50.721484 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:50.721338 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" event={"ID":"64468005-61ea-4f2a-b72c-9c30c6a4d686","Type":"ContainerStarted","Data":"782a2d893db6bb620cd47450c710e2e31bf94afb9d5989a6a97f1f631396990d"} Oct 30 12:59:50.769043 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:50.768959 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" podStartSLOduration=1.768945824 podStartE2EDuration="1.768945824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:50.767139069 +0000 UTC m=+889.370637515" watchObservedRunningTime="2026-10-30 12:59:50.768945824 +0000 UTC m=+889.372444269" Oct 30 12:59:51.192222 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:51.192194 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 12:59:55.366735 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:55.366696 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj"] Oct 30 12:59:55.377791 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:55.377767 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 12:59:55.395947 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:55.395918 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj"] Oct 30 12:59:55.406395 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:55.406365 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-7a3603\"/\"openshift-service-ca.crt\"" Oct 30 12:59:55.406537 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:55.406463 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-7a3603\"/\"premium-w5-7a3603-kserve-self-signed-certs\"" Oct 30 12:59:55.406633 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:55.406615 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-7a3603\"/\"kube-root-ca.crt\"" Oct 30 12:59:55.408419 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:55.408400 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-7a3603\"/\"default-dockercfg-vbp7s\"" Oct 30 12:59:55.515072 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:55.515029 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/089cb620-913d-47c7-b8f7-9b2298c194bf-model-cache\") pod \"premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj\" (UID: \"089cb620-913d-47c7-b8f7-9b2298c194bf\") " pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 12:59:55.515255 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:55.515086 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtjzs\" (UniqueName: \"kubernetes.io/projected/089cb620-913d-47c7-b8f7-9b2298c194bf-kube-api-access-wtjzs\") pod \"premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj\" (UID: \"089cb620-913d-47c7-b8f7-9b2298c194bf\") " pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 12:59:55.515255 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:55.515204 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/089cb620-913d-47c7-b8f7-9b2298c194bf-tmp-dir\") pod \"premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj\" (UID: \"089cb620-913d-47c7-b8f7-9b2298c194bf\") " pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 12:59:55.515382 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:55.515311 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/089cb620-913d-47c7-b8f7-9b2298c194bf-home\") pod \"premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj\" (UID: \"089cb620-913d-47c7-b8f7-9b2298c194bf\") " pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 12:59:55.515382 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:55.515353 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/089cb620-913d-47c7-b8f7-9b2298c194bf-dshm\") pod \"premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj\" (UID: \"089cb620-913d-47c7-b8f7-9b2298c194bf\") " pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 12:59:55.515503 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:55.515481 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/089cb620-913d-47c7-b8f7-9b2298c194bf-tls-certs\") pod \"premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj\" (UID: \"089cb620-913d-47c7-b8f7-9b2298c194bf\") " pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 12:59:55.616053 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:55.616022 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/089cb620-913d-47c7-b8f7-9b2298c194bf-model-cache\") pod \"premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj\" (UID: \"089cb620-913d-47c7-b8f7-9b2298c194bf\") " pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 12:59:55.616053 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:55.616055 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtjzs\" (UniqueName: \"kubernetes.io/projected/089cb620-913d-47c7-b8f7-9b2298c194bf-kube-api-access-wtjzs\") pod \"premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj\" (UID: \"089cb620-913d-47c7-b8f7-9b2298c194bf\") " pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 12:59:55.616350 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:55.616215 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/089cb620-913d-47c7-b8f7-9b2298c194bf-tmp-dir\") pod \"premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj\" (UID: \"089cb620-913d-47c7-b8f7-9b2298c194bf\") " pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 12:59:55.616350 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:55.616317 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/089cb620-913d-47c7-b8f7-9b2298c194bf-home\") pod \"premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj\" (UID: \"089cb620-913d-47c7-b8f7-9b2298c194bf\") " pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 12:59:55.616468 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:55.616360 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/089cb620-913d-47c7-b8f7-9b2298c194bf-dshm\") pod \"premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj\" (UID: \"089cb620-913d-47c7-b8f7-9b2298c194bf\") " pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 12:59:55.616522 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:55.616506 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/089cb620-913d-47c7-b8f7-9b2298c194bf-tls-certs\") pod \"premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj\" (UID: \"089cb620-913d-47c7-b8f7-9b2298c194bf\") " pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 12:59:55.616712 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:55.616685 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/089cb620-913d-47c7-b8f7-9b2298c194bf-tmp-dir\") pod \"premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj\" (UID: \"089cb620-913d-47c7-b8f7-9b2298c194bf\") " pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 12:59:55.616831 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:55.616770 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/089cb620-913d-47c7-b8f7-9b2298c194bf-home\") pod \"premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj\" (UID: \"089cb620-913d-47c7-b8f7-9b2298c194bf\") " pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 12:59:55.616933 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:55.616911 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/089cb620-913d-47c7-b8f7-9b2298c194bf-model-cache\") pod \"premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj\" (UID: \"089cb620-913d-47c7-b8f7-9b2298c194bf\") " pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 12:59:55.618774 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:55.618753 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/089cb620-913d-47c7-b8f7-9b2298c194bf-dshm\") pod \"premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj\" (UID: \"089cb620-913d-47c7-b8f7-9b2298c194bf\") " pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 12:59:55.619138 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:55.619117 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/089cb620-913d-47c7-b8f7-9b2298c194bf-tls-certs\") pod \"premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj\" (UID: \"089cb620-913d-47c7-b8f7-9b2298c194bf\") " pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 12:59:55.620862 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:55.620847 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" Oct 30 12:59:55.634857 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:55.634834 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" Oct 30 12:59:55.640819 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:55.640797 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtjzs\" (UniqueName: \"kubernetes.io/projected/089cb620-913d-47c7-b8f7-9b2298c194bf-kube-api-access-wtjzs\") pod \"premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj\" (UID: \"089cb620-913d-47c7-b8f7-9b2298c194bf\") " pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 12:59:55.691507 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:55.691476 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 12:59:55.770563 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:55.770534 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" Oct 30 12:59:55.997708 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:55.997684 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj"] Oct 30 12:59:55.999856 ip-10-0-142-194 kubenswrapper[4931]: W1030 12:59:55.999824 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod089cb620_913d_47c7_b8f7_9b2298c194bf.slice/crio-6f7e1c7bbc7b3fafe096727bd0c94a4dcbf58ced4641102a68a0fe0dbe1a06f9 WatchSource:0}: Error finding container 6f7e1c7bbc7b3fafe096727bd0c94a4dcbf58ced4641102a68a0fe0dbe1a06f9: Status 404 returned error can't find the container with id 6f7e1c7bbc7b3fafe096727bd0c94a4dcbf58ced4641102a68a0fe0dbe1a06f9 Oct 30 12:59:56.763467 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:56.763429 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" event={"ID":"089cb620-913d-47c7-b8f7-9b2298c194bf","Type":"ContainerStarted","Data":"0f23ba70390bbf81ed1106de56905c391ea0ded209bfc6be651ba6be9e4dd4ba"} Oct 30 12:59:56.763975 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:56.763950 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" event={"ID":"089cb620-913d-47c7-b8f7-9b2298c194bf","Type":"ContainerStarted","Data":"6f7e1c7bbc7b3fafe096727bd0c94a4dcbf58ced4641102a68a0fe0dbe1a06f9"} Oct 30 12:59:56.810052 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:56.810002 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" podStartSLOduration=1.809990019 podStartE2EDuration="1.809990019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:56.808183101 +0000 UTC m=+895.411681548" watchObservedRunningTime="2026-10-30 12:59:56.809990019 +0000 UTC m=+895.413488514" Oct 30 12:59:57.692686 ip-10-0-142-194 kubenswrapper[4931]: I1030 12:59:57.692620 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 13:00:00.187775 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.187742 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8"] Oct 30 13:00:00.191596 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.191573 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv"] Oct 30 13:00:00.191754 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.191704 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8" Oct 30 13:00:00.196400 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.196378 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 13:00:00.196666 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.196646 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv" Oct 30 13:00:00.208539 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.208518 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 13:00:00.215147 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.215124 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng"] Oct 30 13:00:00.218665 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.218644 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng" Oct 30 13:00:00.221304 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.221284 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8"] Oct 30 13:00:00.224988 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.224372 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv"] Oct 30 13:00:00.230766 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.230741 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889420-r2wml"] Oct 30 13:00:00.236918 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.236894 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-r2wml" Oct 30 13:00:00.265027 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.264938 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng"] Oct 30 13:00:00.267962 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.267934 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889420-r2wml"] Oct 30 13:00:00.292058 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.292031 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:00:00.367221 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.367186 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/995b547e-ad4b-46a1-b82c-f29212e99505-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8\" (UID: \"995b547e-ad4b-46a1-b82c-f29212e99505\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8" Oct 30 13:00:00.367446 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.367227 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhnvb\" (UniqueName: \"kubernetes.io/projected/995b547e-ad4b-46a1-b82c-f29212e99505-kube-api-access-qhnvb\") pod \"maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8\" (UID: \"995b547e-ad4b-46a1-b82c-f29212e99505\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8" Oct 30 13:00:00.367446 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.367280 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bd13191-96b6-414b-be8c-91cb2cc46035-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng\" (UID: \"4bd13191-96b6-414b-be8c-91cb2cc46035\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng" Oct 30 13:00:00.367446 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.367412 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhcgn\" (UniqueName: \"kubernetes.io/projected/58fed038-5b34-4f03-bdf3-99d8e6aa830c-kube-api-access-fhcgn\") pod \"maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv\" (UID: \"58fed038-5b34-4f03-bdf3-99d8e6aa830c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv" Oct 30 13:00:00.367446 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.367440 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwnvm\" (UniqueName: \"kubernetes.io/projected/4bd13191-96b6-414b-be8c-91cb2cc46035-kube-api-access-pwnvm\") pod \"maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng\" (UID: \"4bd13191-96b6-414b-be8c-91cb2cc46035\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng" Oct 30 13:00:00.367658 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.367458 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkpsm\" (UniqueName: \"kubernetes.io/projected/42de5f66-15b6-4383-801e-963988e0ecc6-kube-api-access-rkpsm\") pod \"maas-api-key-cleanup-29889420-r2wml\" (UID: \"42de5f66-15b6-4383-801e-963988e0ecc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-r2wml" Oct 30 13:00:00.367693 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.367671 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42de5f66-15b6-4383-801e-963988e0ecc6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889420-r2wml\" (UID: \"42de5f66-15b6-4383-801e-963988e0ecc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-r2wml" Oct 30 13:00:00.367775 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.367760 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58fed038-5b34-4f03-bdf3-99d8e6aa830c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv\" (UID: \"58fed038-5b34-4f03-bdf3-99d8e6aa830c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv" Oct 30 13:00:00.468938 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.468899 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42de5f66-15b6-4383-801e-963988e0ecc6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889420-r2wml\" (UID: \"42de5f66-15b6-4383-801e-963988e0ecc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-r2wml" Oct 30 13:00:00.469143 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.468951 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58fed038-5b34-4f03-bdf3-99d8e6aa830c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv\" (UID: \"58fed038-5b34-4f03-bdf3-99d8e6aa830c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv" Oct 30 13:00:00.469143 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.468983 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/995b547e-ad4b-46a1-b82c-f29212e99505-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8\" (UID: \"995b547e-ad4b-46a1-b82c-f29212e99505\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8" Oct 30 13:00:00.469143 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.469006 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhnvb\" (UniqueName: \"kubernetes.io/projected/995b547e-ad4b-46a1-b82c-f29212e99505-kube-api-access-qhnvb\") pod \"maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8\" (UID: \"995b547e-ad4b-46a1-b82c-f29212e99505\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8" Oct 30 13:00:00.469143 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.469037 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bd13191-96b6-414b-be8c-91cb2cc46035-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng\" (UID: \"4bd13191-96b6-414b-be8c-91cb2cc46035\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng" Oct 30 13:00:00.469143 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.469072 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhcgn\" (UniqueName: \"kubernetes.io/projected/58fed038-5b34-4f03-bdf3-99d8e6aa830c-kube-api-access-fhcgn\") pod \"maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv\" (UID: \"58fed038-5b34-4f03-bdf3-99d8e6aa830c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv" Oct 30 13:00:00.469143 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.469109 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwnvm\" (UniqueName: \"kubernetes.io/projected/4bd13191-96b6-414b-be8c-91cb2cc46035-kube-api-access-pwnvm\") pod \"maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng\" (UID: \"4bd13191-96b6-414b-be8c-91cb2cc46035\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng" Oct 30 13:00:00.469498 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.469152 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkpsm\" (UniqueName: \"kubernetes.io/projected/42de5f66-15b6-4383-801e-963988e0ecc6-kube-api-access-rkpsm\") pod \"maas-api-key-cleanup-29889420-r2wml\" (UID: \"42de5f66-15b6-4383-801e-963988e0ecc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-r2wml" Oct 30 13:00:00.469859 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.469834 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42de5f66-15b6-4383-801e-963988e0ecc6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889420-r2wml\" (UID: \"42de5f66-15b6-4383-801e-963988e0ecc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-r2wml" Oct 30 13:00:00.469954 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.469836 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58fed038-5b34-4f03-bdf3-99d8e6aa830c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv\" (UID: \"58fed038-5b34-4f03-bdf3-99d8e6aa830c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv" Oct 30 13:00:00.469954 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.469842 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bd13191-96b6-414b-be8c-91cb2cc46035-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng\" (UID: \"4bd13191-96b6-414b-be8c-91cb2cc46035\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng" Oct 30 13:00:00.470120 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.470099 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/995b547e-ad4b-46a1-b82c-f29212e99505-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8\" (UID: \"995b547e-ad4b-46a1-b82c-f29212e99505\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8" Oct 30 13:00:00.494098 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.494065 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwnvm\" (UniqueName: \"kubernetes.io/projected/4bd13191-96b6-414b-be8c-91cb2cc46035-kube-api-access-pwnvm\") pod \"maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng\" (UID: \"4bd13191-96b6-414b-be8c-91cb2cc46035\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng" Oct 30 13:00:00.494098 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.494084 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhcgn\" (UniqueName: \"kubernetes.io/projected/58fed038-5b34-4f03-bdf3-99d8e6aa830c-kube-api-access-fhcgn\") pod \"maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv\" (UID: \"58fed038-5b34-4f03-bdf3-99d8e6aa830c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv" Oct 30 13:00:00.494317 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.494137 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhnvb\" (UniqueName: \"kubernetes.io/projected/995b547e-ad4b-46a1-b82c-f29212e99505-kube-api-access-qhnvb\") pod \"maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8\" (UID: \"995b547e-ad4b-46a1-b82c-f29212e99505\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8" Oct 30 13:00:00.496641 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.496619 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkpsm\" (UniqueName: \"kubernetes.io/projected/42de5f66-15b6-4383-801e-963988e0ecc6-kube-api-access-rkpsm\") pod \"maas-api-key-cleanup-29889420-r2wml\" (UID: \"42de5f66-15b6-4383-801e-963988e0ecc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-r2wml" Oct 30 13:00:00.507698 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.507579 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8" Oct 30 13:00:00.515066 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.515046 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv" Oct 30 13:00:00.540586 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.540555 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng" Oct 30 13:00:00.551967 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.551732 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-r2wml" Oct 30 13:00:00.589728 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.589387 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2"] Oct 30 13:00:00.604071 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.602859 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:00:00.624318 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.624288 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2"] Oct 30 13:00:00.625365 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.624792 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-09db16\"/\"kube-root-ca.crt\"" Oct 30 13:00:00.625703 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.624895 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-09db16\"/\"default-dockercfg-xg4df\"" Oct 30 13:00:00.625851 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.624946 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-09db16\"/\"openshift-service-ca.crt\"" Oct 30 13:00:00.625990 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.625059 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-09db16\"/\"premium-w3-09db16-kserve-self-signed-certs\"" Oct 30 13:00:00.773670 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.773579 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65799c14-8de4-4b4a-90df-7cd257e1e581-dshm\") pod \"premium-w3-09db16-kserve-59f68cb797-zjvc2\" (UID: \"65799c14-8de4-4b4a-90df-7cd257e1e581\") " pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:00:00.773835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.773672 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47zqt\" (UniqueName: \"kubernetes.io/projected/65799c14-8de4-4b4a-90df-7cd257e1e581-kube-api-access-47zqt\") pod \"premium-w3-09db16-kserve-59f68cb797-zjvc2\" (UID: \"65799c14-8de4-4b4a-90df-7cd257e1e581\") " pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:00:00.773835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.773714 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65799c14-8de4-4b4a-90df-7cd257e1e581-tmp-dir\") pod \"premium-w3-09db16-kserve-59f68cb797-zjvc2\" (UID: \"65799c14-8de4-4b4a-90df-7cd257e1e581\") " pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:00:00.773835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.773754 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65799c14-8de4-4b4a-90df-7cd257e1e581-model-cache\") pod \"premium-w3-09db16-kserve-59f68cb797-zjvc2\" (UID: \"65799c14-8de4-4b4a-90df-7cd257e1e581\") " pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:00:00.774024 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.773844 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65799c14-8de4-4b4a-90df-7cd257e1e581-home\") pod \"premium-w3-09db16-kserve-59f68cb797-zjvc2\" (UID: \"65799c14-8de4-4b4a-90df-7cd257e1e581\") " pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:00:00.774024 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.773864 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65799c14-8de4-4b4a-90df-7cd257e1e581-tls-certs\") pod \"premium-w3-09db16-kserve-59f68cb797-zjvc2\" (UID: \"65799c14-8de4-4b4a-90df-7cd257e1e581\") " pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:00:00.818797 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.818752 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 13:00:00.828706 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.828682 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8"] Oct 30 13:00:00.830299 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:00:00.830255 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod995b547e_ad4b_46a1_b82c_f29212e99505.slice/crio-dd853c587a3c9b891e87242ff0ac5ebfc543d4b84176d38c17461464ab1dbc6a WatchSource:0}: Error finding container dd853c587a3c9b891e87242ff0ac5ebfc543d4b84176d38c17461464ab1dbc6a: Status 404 returned error can't find the container with id dd853c587a3c9b891e87242ff0ac5ebfc543d4b84176d38c17461464ab1dbc6a Oct 30 13:00:00.851992 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.851965 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv"] Oct 30 13:00:00.853300 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:00:00.853245 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58fed038_5b34_4f03_bdf3_99d8e6aa830c.slice/crio-94d75278435c7c6ee18941b99483a4d2c1f8950a5ad27420c429f810e49e46d3 WatchSource:0}: Error finding container 94d75278435c7c6ee18941b99483a4d2c1f8950a5ad27420c429f810e49e46d3: Status 404 returned error can't find the container with id 94d75278435c7c6ee18941b99483a4d2c1f8950a5ad27420c429f810e49e46d3 Oct 30 13:00:00.874989 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.874960 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65799c14-8de4-4b4a-90df-7cd257e1e581-home\") pod \"premium-w3-09db16-kserve-59f68cb797-zjvc2\" (UID: \"65799c14-8de4-4b4a-90df-7cd257e1e581\") " pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:00:00.875167 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.874995 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65799c14-8de4-4b4a-90df-7cd257e1e581-tls-certs\") pod \"premium-w3-09db16-kserve-59f68cb797-zjvc2\" (UID: \"65799c14-8de4-4b4a-90df-7cd257e1e581\") " pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:00:00.875167 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.875032 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65799c14-8de4-4b4a-90df-7cd257e1e581-dshm\") pod \"premium-w3-09db16-kserve-59f68cb797-zjvc2\" (UID: \"65799c14-8de4-4b4a-90df-7cd257e1e581\") " pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:00:00.875167 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.875069 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47zqt\" (UniqueName: \"kubernetes.io/projected/65799c14-8de4-4b4a-90df-7cd257e1e581-kube-api-access-47zqt\") pod \"premium-w3-09db16-kserve-59f68cb797-zjvc2\" (UID: \"65799c14-8de4-4b4a-90df-7cd257e1e581\") " pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:00:00.875167 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.875108 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65799c14-8de4-4b4a-90df-7cd257e1e581-tmp-dir\") pod \"premium-w3-09db16-kserve-59f68cb797-zjvc2\" (UID: \"65799c14-8de4-4b4a-90df-7cd257e1e581\") " pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:00:00.875167 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.875148 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65799c14-8de4-4b4a-90df-7cd257e1e581-model-cache\") pod \"premium-w3-09db16-kserve-59f68cb797-zjvc2\" (UID: \"65799c14-8de4-4b4a-90df-7cd257e1e581\") " pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:00:00.875519 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.875492 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65799c14-8de4-4b4a-90df-7cd257e1e581-home\") pod \"premium-w3-09db16-kserve-59f68cb797-zjvc2\" (UID: \"65799c14-8de4-4b4a-90df-7cd257e1e581\") " pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:00:00.875607 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.875564 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65799c14-8de4-4b4a-90df-7cd257e1e581-model-cache\") pod \"premium-w3-09db16-kserve-59f68cb797-zjvc2\" (UID: \"65799c14-8de4-4b4a-90df-7cd257e1e581\") " pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:00:00.875729 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.875705 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65799c14-8de4-4b4a-90df-7cd257e1e581-tmp-dir\") pod \"premium-w3-09db16-kserve-59f68cb797-zjvc2\" (UID: \"65799c14-8de4-4b4a-90df-7cd257e1e581\") " pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:00:00.878417 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.878002 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65799c14-8de4-4b4a-90df-7cd257e1e581-dshm\") pod \"premium-w3-09db16-kserve-59f68cb797-zjvc2\" (UID: \"65799c14-8de4-4b4a-90df-7cd257e1e581\") " pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:00:00.878562 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.878448 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65799c14-8de4-4b4a-90df-7cd257e1e581-tls-certs\") pod \"premium-w3-09db16-kserve-59f68cb797-zjvc2\" (UID: \"65799c14-8de4-4b4a-90df-7cd257e1e581\") " pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:00:00.887461 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:00:00.887429 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd13191_96b6_414b_be8c_91cb2cc46035.slice/crio-da3f61f39628680d1467def0e899307dd72de30f8b191f8c2f16b496b94a3f4d WatchSource:0}: Error finding container da3f61f39628680d1467def0e899307dd72de30f8b191f8c2f16b496b94a3f4d: Status 404 returned error can't find the container with id da3f61f39628680d1467def0e899307dd72de30f8b191f8c2f16b496b94a3f4d Oct 30 13:00:00.891691 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.891664 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng"] Oct 30 13:00:00.900313 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.900254 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47zqt\" (UniqueName: \"kubernetes.io/projected/65799c14-8de4-4b4a-90df-7cd257e1e581-kube-api-access-47zqt\") pod \"premium-w3-09db16-kserve-59f68cb797-zjvc2\" (UID: \"65799c14-8de4-4b4a-90df-7cd257e1e581\") " pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:00:00.906581 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.906556 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889420-r2wml"] Oct 30 13:00:00.908381 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:00:00.908356 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42de5f66_15b6_4383_801e_963988e0ecc6.slice/crio-be410a7fe69a48c0331c2c17d6757b408cb01444b3a6217b6f9b9160c932e721 WatchSource:0}: Error finding container be410a7fe69a48c0331c2c17d6757b408cb01444b3a6217b6f9b9160c932e721: Status 404 returned error can't find the container with id be410a7fe69a48c0331c2c17d6757b408cb01444b3a6217b6f9b9160c932e721 Oct 30 13:00:00.937591 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:00.937566 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:00:01.248720 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:01.248692 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2"] Oct 30 13:00:01.250128 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:00:01.250096 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65799c14_8de4_4b4a_90df_7cd257e1e581.slice/crio-580a6913d9ddc92ff6ec1dc0e8d9c8da8006c1acfbb44c4304a06c64c2ab88f8 WatchSource:0}: Error finding container 580a6913d9ddc92ff6ec1dc0e8d9c8da8006c1acfbb44c4304a06c64c2ab88f8: Status 404 returned error can't find the container with id 580a6913d9ddc92ff6ec1dc0e8d9c8da8006c1acfbb44c4304a06c64c2ab88f8 Oct 30 13:00:01.601699 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:01.601617 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/2.log" Oct 30 13:00:01.603290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:01.603235 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/2.log" Oct 30 13:00:01.813310 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:01.813252 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv" event={"ID":"58fed038-5b34-4f03-bdf3-99d8e6aa830c","Type":"ContainerStarted","Data":"94d75278435c7c6ee18941b99483a4d2c1f8950a5ad27420c429f810e49e46d3"} Oct 30 13:00:01.814596 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:01.814564 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng" event={"ID":"4bd13191-96b6-414b-be8c-91cb2cc46035","Type":"ContainerStarted","Data":"da3f61f39628680d1467def0e899307dd72de30f8b191f8c2f16b496b94a3f4d"} Oct 30 13:00:01.815801 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:01.815777 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8" event={"ID":"995b547e-ad4b-46a1-b82c-f29212e99505","Type":"ContainerStarted","Data":"dd853c587a3c9b891e87242ff0ac5ebfc543d4b84176d38c17461464ab1dbc6a"} Oct 30 13:00:01.817243 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:01.817210 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" event={"ID":"65799c14-8de4-4b4a-90df-7cd257e1e581","Type":"ContainerStarted","Data":"d97fc7e4e5cbc5ef1300d36aa84ea9e353aaec9f6d23bbeb29a96baab879d54e"} Oct 30 13:00:01.817359 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:01.817249 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" event={"ID":"65799c14-8de4-4b4a-90df-7cd257e1e581","Type":"ContainerStarted","Data":"580a6913d9ddc92ff6ec1dc0e8d9c8da8006c1acfbb44c4304a06c64c2ab88f8"} Oct 30 13:00:01.818492 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:01.818468 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-r2wml" event={"ID":"42de5f66-15b6-4383-801e-963988e0ecc6","Type":"ContainerStarted","Data":"be410a7fe69a48c0331c2c17d6757b408cb01444b3a6217b6f9b9160c932e721"} Oct 30 13:00:01.871960 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:01.871853 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" podStartSLOduration=1.871837564 podStartE2EDuration="1.871837564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:01.869896611 +0000 UTC m=+900.473395056" watchObservedRunningTime="2026-10-30 13:00:01.871837564 +0000 UTC m=+900.475336029" Oct 30 13:00:01.938244 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:01.938208 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:00:04.619452 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:04.619378 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:00:04.841646 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:04.841604 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv" event={"ID":"58fed038-5b34-4f03-bdf3-99d8e6aa830c","Type":"ContainerStarted","Data":"1d8fe0a06690c137db431dbb41310847041c7313dccb760fd239c8b00670ae64"} Oct 30 13:00:04.843324 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:04.843277 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng" event={"ID":"4bd13191-96b6-414b-be8c-91cb2cc46035","Type":"ContainerStarted","Data":"7f1e48b6e053e1bf94ea19c497da1b45f38d9ce6c62fbfce73c715eef859849b"} Oct 30 13:00:04.844860 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:04.844832 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8" event={"ID":"995b547e-ad4b-46a1-b82c-f29212e99505","Type":"ContainerStarted","Data":"528845ac889caeda49962a156f308a9166ff6328546a257ce341e1ffcf200508"} Oct 30 13:00:04.891716 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:04.891611 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv" podStartSLOduration=1.427959523 podStartE2EDuration="4.89159513s" totalImagesPullingTime="3.463635607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:04.888841897 +0000 UTC m=+903.492340340" watchObservedRunningTime="2026-10-30 13:00:04.89159513 +0000 UTC m=+903.495093576" Oct 30 13:00:04.936956 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:04.936906 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8" podStartSLOduration=1.454969379 podStartE2EDuration="4.936892161s" totalImagesPullingTime="3.481922782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:04.935387651 +0000 UTC m=+903.538886096" watchObservedRunningTime="2026-10-30 13:00:04.936892161 +0000 UTC m=+903.540390609" Oct 30 13:00:04.984922 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:04.984868 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng" podStartSLOduration=1.4797374890000001 podStartE2EDuration="4.984853363s" totalImagesPullingTime="3.505115874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:04.982423013 +0000 UTC m=+903.585921459" watchObservedRunningTime="2026-10-30 13:00:04.984853363 +0000 UTC m=+903.588351808" Oct 30 13:00:05.692232 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:05.692191 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 13:00:05.709982 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:05.709951 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 13:00:05.855181 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:05.855085 4931 generic.go:356] "Generic (PLEG): container finished" podID="58fed038-5b34-4f03-bdf3-99d8e6aa830c" containerID="1d8fe0a06690c137db431dbb41310847041c7313dccb760fd239c8b00670ae64" exitCode=60 Oct 30 13:00:05.855384 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:05.855176 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv" event={"ID":"58fed038-5b34-4f03-bdf3-99d8e6aa830c","Type":"ContainerDied","Data":"1d8fe0a06690c137db431dbb41310847041c7313dccb760fd239c8b00670ae64"} Oct 30 13:00:05.855498 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:05.855475 4931 scope.go:122] "RemoveContainer" containerID="1d8fe0a06690c137db431dbb41310847041c7313dccb760fd239c8b00670ae64" Oct 30 13:00:05.857497 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:05.857473 4931 generic.go:356] "Generic (PLEG): container finished" podID="4bd13191-96b6-414b-be8c-91cb2cc46035" containerID="7f1e48b6e053e1bf94ea19c497da1b45f38d9ce6c62fbfce73c715eef859849b" exitCode=60 Oct 30 13:00:05.857618 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:05.857528 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng" event={"ID":"4bd13191-96b6-414b-be8c-91cb2cc46035","Type":"ContainerDied","Data":"7f1e48b6e053e1bf94ea19c497da1b45f38d9ce6c62fbfce73c715eef859849b"} Oct 30 13:00:05.857808 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:05.857783 4931 scope.go:122] "RemoveContainer" containerID="7f1e48b6e053e1bf94ea19c497da1b45f38d9ce6c62fbfce73c715eef859849b" Oct 30 13:00:05.860390 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:05.860336 4931 generic.go:356] "Generic (PLEG): container finished" podID="995b547e-ad4b-46a1-b82c-f29212e99505" containerID="528845ac889caeda49962a156f308a9166ff6328546a257ce341e1ffcf200508" exitCode=60 Oct 30 13:00:05.860637 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:05.860487 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8" event={"ID":"995b547e-ad4b-46a1-b82c-f29212e99505","Type":"ContainerDied","Data":"528845ac889caeda49962a156f308a9166ff6328546a257ce341e1ffcf200508"} Oct 30 13:00:05.860637 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:05.860625 4931 scope.go:122] "RemoveContainer" containerID="528845ac889caeda49962a156f308a9166ff6328546a257ce341e1ffcf200508" Oct 30 13:00:05.872067 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:05.870732 4931 generic.go:356] "Generic (PLEG): container finished" podID="42de5f66-15b6-4383-801e-963988e0ecc6" containerID="e207ca56a1b6074f527b574769c84c74b7e13b174e7024c4c5221532105c7e44" exitCode=60 Oct 30 13:00:05.872067 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:05.870848 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-r2wml" event={"ID":"42de5f66-15b6-4383-801e-963988e0ecc6","Type":"ContainerDied","Data":"e207ca56a1b6074f527b574769c84c74b7e13b174e7024c4c5221532105c7e44"} Oct 30 13:00:05.872067 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:05.871100 4931 scope.go:122] "RemoveContainer" containerID="e207ca56a1b6074f527b574769c84c74b7e13b174e7024c4c5221532105c7e44" Oct 30 13:00:05.889631 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:05.889612 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 13:00:06.507751 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:06.507716 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k"] Oct 30 13:00:06.512023 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:06.511996 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:00:06.528326 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:06.528300 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-dd90cb\"/\"premium-w6-dd90cb-kserve-self-signed-certs\"" Oct 30 13:00:06.541887 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:06.541859 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k"] Oct 30 13:00:06.549187 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:06.549151 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-tmp-dir\") pod \"premium-w6-dd90cb-kserve-5fd959df56-7rz9k\" (UID: \"e52bb4f7-ceff-4e39-91ce-a8f48b205a0a\") " pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:00:06.549346 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:06.549202 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxhpb\" (UniqueName: \"kubernetes.io/projected/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-kube-api-access-kxhpb\") pod \"premium-w6-dd90cb-kserve-5fd959df56-7rz9k\" (UID: \"e52bb4f7-ceff-4e39-91ce-a8f48b205a0a\") " pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:00:06.549346 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:06.549235 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-tls-certs\") pod \"premium-w6-dd90cb-kserve-5fd959df56-7rz9k\" (UID: \"e52bb4f7-ceff-4e39-91ce-a8f48b205a0a\") " pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:00:06.549346 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:06.549278 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-home\") pod \"premium-w6-dd90cb-kserve-5fd959df56-7rz9k\" (UID: \"e52bb4f7-ceff-4e39-91ce-a8f48b205a0a\") " pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:00:06.549534 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:06.549367 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-dshm\") pod \"premium-w6-dd90cb-kserve-5fd959df56-7rz9k\" (UID: \"e52bb4f7-ceff-4e39-91ce-a8f48b205a0a\") " pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:00:06.549534 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:06.549394 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-model-cache\") pod \"premium-w6-dd90cb-kserve-5fd959df56-7rz9k\" (UID: \"e52bb4f7-ceff-4e39-91ce-a8f48b205a0a\") " pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:00:06.652905 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:06.651189 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-tmp-dir\") pod \"premium-w6-dd90cb-kserve-5fd959df56-7rz9k\" (UID: \"e52bb4f7-ceff-4e39-91ce-a8f48b205a0a\") " pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:00:06.652905 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:06.651239 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxhpb\" (UniqueName: \"kubernetes.io/projected/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-kube-api-access-kxhpb\") pod \"premium-w6-dd90cb-kserve-5fd959df56-7rz9k\" (UID: \"e52bb4f7-ceff-4e39-91ce-a8f48b205a0a\") " pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:00:06.652905 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:06.651288 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-tls-certs\") pod \"premium-w6-dd90cb-kserve-5fd959df56-7rz9k\" (UID: \"e52bb4f7-ceff-4e39-91ce-a8f48b205a0a\") " pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:00:06.652905 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:06.651316 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-home\") pod \"premium-w6-dd90cb-kserve-5fd959df56-7rz9k\" (UID: \"e52bb4f7-ceff-4e39-91ce-a8f48b205a0a\") " pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:00:06.652905 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:06.651395 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-dshm\") pod \"premium-w6-dd90cb-kserve-5fd959df56-7rz9k\" (UID: \"e52bb4f7-ceff-4e39-91ce-a8f48b205a0a\") " pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:00:06.652905 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:06.651437 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-model-cache\") pod \"premium-w6-dd90cb-kserve-5fd959df56-7rz9k\" (UID: \"e52bb4f7-ceff-4e39-91ce-a8f48b205a0a\") " pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:00:06.652905 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:06.651934 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-model-cache\") pod \"premium-w6-dd90cb-kserve-5fd959df56-7rz9k\" (UID: \"e52bb4f7-ceff-4e39-91ce-a8f48b205a0a\") " pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:00:06.652905 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:06.652169 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-tmp-dir\") pod \"premium-w6-dd90cb-kserve-5fd959df56-7rz9k\" (UID: \"e52bb4f7-ceff-4e39-91ce-a8f48b205a0a\") " pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:00:06.654081 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:06.654060 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-home\") pod \"premium-w6-dd90cb-kserve-5fd959df56-7rz9k\" (UID: \"e52bb4f7-ceff-4e39-91ce-a8f48b205a0a\") " pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:00:06.655970 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:06.655922 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-tls-certs\") pod \"premium-w6-dd90cb-kserve-5fd959df56-7rz9k\" (UID: \"e52bb4f7-ceff-4e39-91ce-a8f48b205a0a\") " pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:00:06.656774 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:06.656733 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-dshm\") pod \"premium-w6-dd90cb-kserve-5fd959df56-7rz9k\" (UID: \"e52bb4f7-ceff-4e39-91ce-a8f48b205a0a\") " pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:00:06.680521 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:06.680492 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxhpb\" (UniqueName: \"kubernetes.io/projected/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-kube-api-access-kxhpb\") pod \"premium-w6-dd90cb-kserve-5fd959df56-7rz9k\" (UID: \"e52bb4f7-ceff-4e39-91ce-a8f48b205a0a\") " pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:00:06.825671 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:06.825592 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:00:06.886670 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:06.881170 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8" event={"ID":"995b547e-ad4b-46a1-b82c-f29212e99505","Type":"ContainerStarted","Data":"b16750a14a8e700e1203d1006c4cef51745245cfa74103a16273fd07e547f266"} Oct 30 13:00:06.886670 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:06.886173 4931 generic.go:356] "Generic (PLEG): container finished" podID="42de5f66-15b6-4383-801e-963988e0ecc6" containerID="7cd37b486d58185b2b61c162caaf62fa4ee52728266c45b105ac741a55c6c365" exitCode=60 Oct 30 13:00:06.886670 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:06.886283 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-r2wml" event={"ID":"42de5f66-15b6-4383-801e-963988e0ecc6","Type":"ContainerDied","Data":"7cd37b486d58185b2b61c162caaf62fa4ee52728266c45b105ac741a55c6c365"} Oct 30 13:00:06.886670 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:06.886321 4931 scope.go:122] "RemoveContainer" containerID="e207ca56a1b6074f527b574769c84c74b7e13b174e7024c4c5221532105c7e44" Oct 30 13:00:06.886670 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:06.886589 4931 scope.go:122] "RemoveContainer" containerID="7cd37b486d58185b2b61c162caaf62fa4ee52728266c45b105ac741a55c6c365" Oct 30 13:00:06.887387 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:00:06.886879 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889420-r2wml_odh-ai-gateway-infra(42de5f66-15b6-4383-801e-963988e0ecc6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-r2wml" podUID="42de5f66-15b6-4383-801e-963988e0ecc6" Oct 30 13:00:06.890654 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:06.890444 4931 generic.go:356] "Generic (PLEG): container finished" podID="58fed038-5b34-4f03-bdf3-99d8e6aa830c" containerID="d32a436ae09f534313cc4a52ce57a5c440376c0fea6e1a9ea261273a7de9dc36" exitCode=60 Oct 30 13:00:06.890753 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:06.890728 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv" event={"ID":"58fed038-5b34-4f03-bdf3-99d8e6aa830c","Type":"ContainerDied","Data":"d32a436ae09f534313cc4a52ce57a5c440376c0fea6e1a9ea261273a7de9dc36"} Oct 30 13:00:06.892562 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:06.890861 4931 scope.go:122] "RemoveContainer" containerID="d32a436ae09f534313cc4a52ce57a5c440376c0fea6e1a9ea261273a7de9dc36" Oct 30 13:00:06.892562 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:00:06.891120 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv_odh-ai-gateway-infra(58fed038-5b34-4f03-bdf3-99d8e6aa830c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv" podUID="58fed038-5b34-4f03-bdf3-99d8e6aa830c" Oct 30 13:00:06.894917 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:06.894755 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng" event={"ID":"4bd13191-96b6-414b-be8c-91cb2cc46035","Type":"ContainerStarted","Data":"93074a78308f742d237708553bc3e385d03407d9152cf3e4fe58ee150fef9b71"} Oct 30 13:00:06.909078 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:06.909039 4931 scope.go:122] "RemoveContainer" containerID="1d8fe0a06690c137db431dbb41310847041c7313dccb760fd239c8b00670ae64" Oct 30 13:00:07.138941 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:07.138912 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k"] Oct 30 13:00:07.140578 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:00:07.140542 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode52bb4f7_ceff_4e39_91ce_a8f48b205a0a.slice/crio-59600529f791ea7a12f67cd90131440f8cc7cdb40720f597e022966a1f9d1a22 WatchSource:0}: Error finding container 59600529f791ea7a12f67cd90131440f8cc7cdb40720f597e022966a1f9d1a22: Status 404 returned error can't find the container with id 59600529f791ea7a12f67cd90131440f8cc7cdb40720f597e022966a1f9d1a22 Oct 30 13:00:07.353048 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:00:07.353009 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:00:07.901434 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:07.901401 4931 generic.go:356] "Generic (PLEG): container finished" podID="995b547e-ad4b-46a1-b82c-f29212e99505" containerID="b16750a14a8e700e1203d1006c4cef51745245cfa74103a16273fd07e547f266" exitCode=60 Oct 30 13:00:07.901898 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:07.901455 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8" event={"ID":"995b547e-ad4b-46a1-b82c-f29212e99505","Type":"ContainerDied","Data":"b16750a14a8e700e1203d1006c4cef51745245cfa74103a16273fd07e547f266"} Oct 30 13:00:07.901898 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:07.901525 4931 scope.go:122] "RemoveContainer" containerID="528845ac889caeda49962a156f308a9166ff6328546a257ce341e1ffcf200508" Oct 30 13:00:07.901898 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:07.901793 4931 scope.go:122] "RemoveContainer" containerID="b16750a14a8e700e1203d1006c4cef51745245cfa74103a16273fd07e547f266" Oct 30 13:00:07.902131 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:00:07.902110 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8_odh-ai-gateway-infra(995b547e-ad4b-46a1-b82c-f29212e99505)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8" podUID="995b547e-ad4b-46a1-b82c-f29212e99505" Oct 30 13:00:07.903309 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:07.903257 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" event={"ID":"e52bb4f7-ceff-4e39-91ce-a8f48b205a0a","Type":"ContainerStarted","Data":"b92efe13a80793dae4efc15df9ee1dfb730fa6366ef1f772b0fcd1b717656636"} Oct 30 13:00:07.903396 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:07.903322 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" event={"ID":"e52bb4f7-ceff-4e39-91ce-a8f48b205a0a","Type":"ContainerStarted","Data":"59600529f791ea7a12f67cd90131440f8cc7cdb40720f597e022966a1f9d1a22"} Oct 30 13:00:07.905064 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:07.905045 4931 scope.go:122] "RemoveContainer" containerID="7cd37b486d58185b2b61c162caaf62fa4ee52728266c45b105ac741a55c6c365" Oct 30 13:00:07.905329 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:00:07.905311 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889420-r2wml_odh-ai-gateway-infra(42de5f66-15b6-4383-801e-963988e0ecc6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-r2wml" podUID="42de5f66-15b6-4383-801e-963988e0ecc6" Oct 30 13:00:07.906593 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:07.906574 4931 scope.go:122] "RemoveContainer" containerID="d32a436ae09f534313cc4a52ce57a5c440376c0fea6e1a9ea261273a7de9dc36" Oct 30 13:00:07.906814 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:00:07.906791 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv_odh-ai-gateway-infra(58fed038-5b34-4f03-bdf3-99d8e6aa830c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv" podUID="58fed038-5b34-4f03-bdf3-99d8e6aa830c" Oct 30 13:00:07.908317 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:07.908295 4931 generic.go:356] "Generic (PLEG): container finished" podID="4bd13191-96b6-414b-be8c-91cb2cc46035" containerID="93074a78308f742d237708553bc3e385d03407d9152cf3e4fe58ee150fef9b71" exitCode=60 Oct 30 13:00:07.908463 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:07.908332 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng" event={"ID":"4bd13191-96b6-414b-be8c-91cb2cc46035","Type":"ContainerDied","Data":"93074a78308f742d237708553bc3e385d03407d9152cf3e4fe58ee150fef9b71"} Oct 30 13:00:07.908611 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:07.908577 4931 scope.go:122] "RemoveContainer" containerID="93074a78308f742d237708553bc3e385d03407d9152cf3e4fe58ee150fef9b71" Oct 30 13:00:07.908788 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:00:07.908769 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng_odh-ai-gateway-infra(4bd13191-96b6-414b-be8c-91cb2cc46035)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng" podUID="4bd13191-96b6-414b-be8c-91cb2cc46035" Oct 30 13:00:07.914368 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:07.914345 4931 scope.go:122] "RemoveContainer" containerID="7f1e48b6e053e1bf94ea19c497da1b45f38d9ce6c62fbfce73c715eef859849b" Oct 30 13:00:08.121579 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:08.121523 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" podStartSLOduration=2.121506131 podStartE2EDuration="2.121506131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:08.118777421 +0000 UTC m=+906.722275866" watchObservedRunningTime="2026-10-30 13:00:08.121506131 +0000 UTC m=+906.725004632" Oct 30 13:00:08.826262 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:08.826223 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:00:08.916311 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:08.916251 4931 scope.go:122] "RemoveContainer" containerID="93074a78308f742d237708553bc3e385d03407d9152cf3e4fe58ee150fef9b71" Oct 30 13:00:08.916765 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:00:08.916571 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng_odh-ai-gateway-infra(4bd13191-96b6-414b-be8c-91cb2cc46035)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng" podUID="4bd13191-96b6-414b-be8c-91cb2cc46035" Oct 30 13:00:08.918215 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:08.918192 4931 scope.go:122] "RemoveContainer" containerID="b16750a14a8e700e1203d1006c4cef51745245cfa74103a16273fd07e547f266" Oct 30 13:00:08.918478 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:00:08.918460 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8_odh-ai-gateway-infra(995b547e-ad4b-46a1-b82c-f29212e99505)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8" podUID="995b547e-ad4b-46a1-b82c-f29212e99505" Oct 30 13:00:10.938386 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:10.938350 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:00:10.951320 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:10.951291 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:00:11.947192 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:11.947161 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:00:16.826662 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:16.826584 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:00:16.839383 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:16.839354 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:00:16.977375 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:16.977344 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:00:17.506240 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:17.506209 4931 scope.go:122] "RemoveContainer" containerID="93074a78308f742d237708553bc3e385d03407d9152cf3e4fe58ee150fef9b71" Oct 30 13:00:17.973512 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:17.973476 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng" event={"ID":"4bd13191-96b6-414b-be8c-91cb2cc46035","Type":"ContainerStarted","Data":"1cf3a4be94eec4b30343844174c86421f1853e22ef2bd3ea9cc692567d376d87"} Oct 30 13:00:18.506562 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:18.506486 4931 scope.go:122] "RemoveContainer" containerID="7cd37b486d58185b2b61c162caaf62fa4ee52728266c45b105ac741a55c6c365" Oct 30 13:00:18.506562 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:18.506552 4931 scope.go:122] "RemoveContainer" containerID="b16750a14a8e700e1203d1006c4cef51745245cfa74103a16273fd07e547f266" Oct 30 13:00:18.983043 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:18.983008 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8" event={"ID":"995b547e-ad4b-46a1-b82c-f29212e99505","Type":"ContainerStarted","Data":"9a56d7f03a6b70cf9d36dbdbafd95cfabb29b61ad4efb870e25d8b063d96fa08"} Oct 30 13:00:18.985100 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:18.985070 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-r2wml" event={"ID":"42de5f66-15b6-4383-801e-963988e0ecc6","Type":"ContainerStarted","Data":"700b91ff0897493ed22fcbf685a0a64bb10ef69b3aadc7a2157f5cbb975be71d"} Oct 30 13:00:18.987160 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:18.987139 4931 generic.go:356] "Generic (PLEG): container finished" podID="4bd13191-96b6-414b-be8c-91cb2cc46035" containerID="1cf3a4be94eec4b30343844174c86421f1853e22ef2bd3ea9cc692567d376d87" exitCode=60 Oct 30 13:00:18.987333 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:18.987165 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng" event={"ID":"4bd13191-96b6-414b-be8c-91cb2cc46035","Type":"ContainerDied","Data":"1cf3a4be94eec4b30343844174c86421f1853e22ef2bd3ea9cc692567d376d87"} Oct 30 13:00:18.987333 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:18.987200 4931 scope.go:122] "RemoveContainer" containerID="93074a78308f742d237708553bc3e385d03407d9152cf3e4fe58ee150fef9b71" Oct 30 13:00:18.987464 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:18.987445 4931 scope.go:122] "RemoveContainer" containerID="1cf3a4be94eec4b30343844174c86421f1853e22ef2bd3ea9cc692567d376d87" Oct 30 13:00:18.987717 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:00:18.987691 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng_odh-ai-gateway-infra(4bd13191-96b6-414b-be8c-91cb2cc46035)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng" podUID="4bd13191-96b6-414b-be8c-91cb2cc46035" Oct 30 13:00:19.049116 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:19.049085 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng"] Oct 30 13:00:19.089718 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:19.089671 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-r2wml" podStartSLOduration=15.237249143 podStartE2EDuration="19.08965649s" totalImagesPullingTime="3.852407347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:19.087568369 +0000 UTC m=+917.691066816" watchObservedRunningTime="2026-10-30 13:00:19.08965649 +0000 UTC m=+917.693154937" Oct 30 13:00:20.005393 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:20.005359 4931 generic.go:356] "Generic (PLEG): container finished" podID="995b547e-ad4b-46a1-b82c-f29212e99505" containerID="9a56d7f03a6b70cf9d36dbdbafd95cfabb29b61ad4efb870e25d8b063d96fa08" exitCode=60 Oct 30 13:00:20.005848 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:20.005400 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8" event={"ID":"995b547e-ad4b-46a1-b82c-f29212e99505","Type":"ContainerDied","Data":"9a56d7f03a6b70cf9d36dbdbafd95cfabb29b61ad4efb870e25d8b063d96fa08"} Oct 30 13:00:20.005848 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:20.005504 4931 scope.go:122] "RemoveContainer" containerID="b16750a14a8e700e1203d1006c4cef51745245cfa74103a16273fd07e547f266" Oct 30 13:00:20.005972 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:20.005864 4931 scope.go:122] "RemoveContainer" containerID="9a56d7f03a6b70cf9d36dbdbafd95cfabb29b61ad4efb870e25d8b063d96fa08" Oct 30 13:00:20.006383 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:00:20.006152 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8_odh-ai-gateway-infra(995b547e-ad4b-46a1-b82c-f29212e99505)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8" podUID="995b547e-ad4b-46a1-b82c-f29212e99505" Oct 30 13:00:20.007836 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:20.007818 4931 generic.go:356] "Generic (PLEG): container finished" podID="42de5f66-15b6-4383-801e-963988e0ecc6" containerID="700b91ff0897493ed22fcbf685a0a64bb10ef69b3aadc7a2157f5cbb975be71d" exitCode=60 Oct 30 13:00:20.007926 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:20.007891 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-r2wml" event={"ID":"42de5f66-15b6-4383-801e-963988e0ecc6","Type":"ContainerDied","Data":"700b91ff0897493ed22fcbf685a0a64bb10ef69b3aadc7a2157f5cbb975be71d"} Oct 30 13:00:20.008202 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:20.008188 4931 scope.go:122] "RemoveContainer" containerID="700b91ff0897493ed22fcbf685a0a64bb10ef69b3aadc7a2157f5cbb975be71d" Oct 30 13:00:20.008467 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:00:20.008449 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889420-r2wml_odh-ai-gateway-infra(42de5f66-15b6-4383-801e-963988e0ecc6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-r2wml" podUID="42de5f66-15b6-4383-801e-963988e0ecc6" Oct 30 13:00:20.018903 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:20.018881 4931 scope.go:122] "RemoveContainer" containerID="7cd37b486d58185b2b61c162caaf62fa4ee52728266c45b105ac741a55c6c365" Oct 30 13:00:20.047321 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:20.047292 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8"] Oct 30 13:00:20.108202 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:20.108170 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889420-r2wml"] Oct 30 13:00:20.177111 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:20.177082 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng" Oct 30 13:00:20.300426 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:20.300331 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bd13191-96b6-414b-be8c-91cb2cc46035-kube-api-access-pwnvm\" (UniqueName: \"kubernetes.io/projected/4bd13191-96b6-414b-be8c-91cb2cc46035-kube-api-access-pwnvm\") pod \"4bd13191-96b6-414b-be8c-91cb2cc46035\" (UID: \"4bd13191-96b6-414b-be8c-91cb2cc46035\") " Oct 30 13:00:20.300426 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:20.300377 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bd13191-96b6-414b-be8c-91cb2cc46035-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bd13191-96b6-414b-be8c-91cb2cc46035-maas-api-service-ca\") pod \"4bd13191-96b6-414b-be8c-91cb2cc46035\" (UID: \"4bd13191-96b6-414b-be8c-91cb2cc46035\") " Oct 30 13:00:20.300754 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:20.300732 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bd13191-96b6-414b-be8c-91cb2cc46035-maas-api-service-ca" pod "4bd13191-96b6-414b-be8c-91cb2cc46035" (UID: "4bd13191-96b6-414b-be8c-91cb2cc46035"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:20.302592 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:20.302569 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bd13191-96b6-414b-be8c-91cb2cc46035-kube-api-access-pwnvm" pod "4bd13191-96b6-414b-be8c-91cb2cc46035" (UID: "4bd13191-96b6-414b-be8c-91cb2cc46035"). InnerVolumeSpecName "kube-api-access-pwnvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:20.402250 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:20.402212 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwnvm\" (UniqueName: \"kubernetes.io/projected/4bd13191-96b6-414b-be8c-91cb2cc46035-kube-api-access-pwnvm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:00:20.402250 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:20.402244 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bd13191-96b6-414b-be8c-91cb2cc46035-maas-api-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:00:21.017639 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:21.017605 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng" Oct 30 13:00:21.017639 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:21.017631 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng" event={"ID":"4bd13191-96b6-414b-be8c-91cb2cc46035","Type":"ContainerDied","Data":"da3f61f39628680d1467def0e899307dd72de30f8b191f8c2f16b496b94a3f4d"} Oct 30 13:00:21.018195 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:21.017692 4931 scope.go:122] "RemoveContainer" containerID="1cf3a4be94eec4b30343844174c86421f1853e22ef2bd3ea9cc692567d376d87" Oct 30 13:00:21.180754 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:21.180725 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8" Oct 30 13:00:21.201362 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:21.201336 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng"] Oct 30 13:00:21.207601 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:21.207577 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889420-q44ng"] Oct 30 13:00:21.216067 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:21.216049 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-r2wml" Oct 30 13:00:21.311935 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:21.311868 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42de5f66-15b6-4383-801e-963988e0ecc6-kube-api-access-rkpsm\" (UniqueName: \"kubernetes.io/projected/42de5f66-15b6-4383-801e-963988e0ecc6-kube-api-access-rkpsm\") pod \"42de5f66-15b6-4383-801e-963988e0ecc6\" (UID: \"42de5f66-15b6-4383-801e-963988e0ecc6\") " Oct 30 13:00:21.311935 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:21.311932 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/995b547e-ad4b-46a1-b82c-f29212e99505-kube-api-access-qhnvb\" (UniqueName: \"kubernetes.io/projected/995b547e-ad4b-46a1-b82c-f29212e99505-kube-api-access-qhnvb\") pod \"995b547e-ad4b-46a1-b82c-f29212e99505\" (UID: \"995b547e-ad4b-46a1-b82c-f29212e99505\") " Oct 30 13:00:21.312152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:21.311991 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42de5f66-15b6-4383-801e-963988e0ecc6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42de5f66-15b6-4383-801e-963988e0ecc6-maas-api-service-ca\") pod \"42de5f66-15b6-4383-801e-963988e0ecc6\" (UID: \"42de5f66-15b6-4383-801e-963988e0ecc6\") " Oct 30 13:00:21.312152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:21.312010 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/995b547e-ad4b-46a1-b82c-f29212e99505-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/995b547e-ad4b-46a1-b82c-f29212e99505-maas-api-service-ca\") pod \"995b547e-ad4b-46a1-b82c-f29212e99505\" (UID: \"995b547e-ad4b-46a1-b82c-f29212e99505\") " Oct 30 13:00:21.312405 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:21.312380 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/995b547e-ad4b-46a1-b82c-f29212e99505-maas-api-service-ca" pod "995b547e-ad4b-46a1-b82c-f29212e99505" (UID: "995b547e-ad4b-46a1-b82c-f29212e99505"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:21.312473 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:21.312387 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42de5f66-15b6-4383-801e-963988e0ecc6-maas-api-service-ca" pod "42de5f66-15b6-4383-801e-963988e0ecc6" (UID: "42de5f66-15b6-4383-801e-963988e0ecc6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:21.314177 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:21.314154 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42de5f66-15b6-4383-801e-963988e0ecc6-kube-api-access-rkpsm" pod "42de5f66-15b6-4383-801e-963988e0ecc6" (UID: "42de5f66-15b6-4383-801e-963988e0ecc6"). InnerVolumeSpecName "kube-api-access-rkpsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:21.314297 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:21.314285 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/995b547e-ad4b-46a1-b82c-f29212e99505-kube-api-access-qhnvb" pod "995b547e-ad4b-46a1-b82c-f29212e99505" (UID: "995b547e-ad4b-46a1-b82c-f29212e99505"). InnerVolumeSpecName "kube-api-access-qhnvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:21.413337 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:21.413303 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkpsm\" (UniqueName: \"kubernetes.io/projected/42de5f66-15b6-4383-801e-963988e0ecc6-kube-api-access-rkpsm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:00:21.413337 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:21.413333 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhnvb\" (UniqueName: \"kubernetes.io/projected/995b547e-ad4b-46a1-b82c-f29212e99505-kube-api-access-qhnvb\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:00:21.413337 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:21.413344 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42de5f66-15b6-4383-801e-963988e0ecc6-maas-api-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:00:21.413564 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:21.413353 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/995b547e-ad4b-46a1-b82c-f29212e99505-maas-api-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:00:21.509745 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:21.509716 4931 scope.go:122] "RemoveContainer" containerID="d32a436ae09f534313cc4a52ce57a5c440376c0fea6e1a9ea261273a7de9dc36" Oct 30 13:00:21.514609 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:21.514586 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bd13191-96b6-414b-be8c-91cb2cc46035" path="/var/lib/kubelet/pods/4bd13191-96b6-414b-be8c-91cb2cc46035/volumes" Oct 30 13:00:21.975605 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:21.975567 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:00:22.036317 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:22.036203 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8" Oct 30 13:00:22.036317 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:22.036234 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8" event={"ID":"995b547e-ad4b-46a1-b82c-f29212e99505","Type":"ContainerDied","Data":"dd853c587a3c9b891e87242ff0ac5ebfc543d4b84176d38c17461464ab1dbc6a"} Oct 30 13:00:22.036864 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:22.036352 4931 scope.go:122] "RemoveContainer" containerID="9a56d7f03a6b70cf9d36dbdbafd95cfabb29b61ad4efb870e25d8b063d96fa08" Oct 30 13:00:22.039060 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:22.039040 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-r2wml" Oct 30 13:00:22.039518 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:22.039499 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-r2wml" event={"ID":"42de5f66-15b6-4383-801e-963988e0ecc6","Type":"ContainerDied","Data":"be410a7fe69a48c0331c2c17d6757b408cb01444b3a6217b6f9b9160c932e721"} Oct 30 13:00:22.041757 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:22.041734 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv" event={"ID":"58fed038-5b34-4f03-bdf3-99d8e6aa830c","Type":"ContainerStarted","Data":"5e70d27b0367c205b27c9f432bcbfc6c116787cd7c9ce8999d2616521260e0bb"} Oct 30 13:00:22.047153 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:22.047122 4931 scope.go:122] "RemoveContainer" containerID="700b91ff0897493ed22fcbf685a0a64bb10ef69b3aadc7a2157f5cbb975be71d" Oct 30 13:00:22.160417 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:22.160385 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889420-r2wml"] Oct 30 13:00:22.194963 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:22.194929 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889420-r2wml"] Oct 30 13:00:22.235343 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:22.235307 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8"] Oct 30 13:00:22.250293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:22.250250 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889420-kwkk8"] Oct 30 13:00:23.052760 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:23.052732 4931 generic.go:356] "Generic (PLEG): container finished" podID="58fed038-5b34-4f03-bdf3-99d8e6aa830c" containerID="5e70d27b0367c205b27c9f432bcbfc6c116787cd7c9ce8999d2616521260e0bb" exitCode=60 Oct 30 13:00:23.053175 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:23.052812 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv" event={"ID":"58fed038-5b34-4f03-bdf3-99d8e6aa830c","Type":"ContainerDied","Data":"5e70d27b0367c205b27c9f432bcbfc6c116787cd7c9ce8999d2616521260e0bb"} Oct 30 13:00:23.053175 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:23.052874 4931 scope.go:122] "RemoveContainer" containerID="d32a436ae09f534313cc4a52ce57a5c440376c0fea6e1a9ea261273a7de9dc36" Oct 30 13:00:23.053175 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:23.053150 4931 scope.go:122] "RemoveContainer" containerID="5e70d27b0367c205b27c9f432bcbfc6c116787cd7c9ce8999d2616521260e0bb" Oct 30 13:00:23.053449 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:00:23.053430 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv_odh-ai-gateway-infra(58fed038-5b34-4f03-bdf3-99d8e6aa830c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv" podUID="58fed038-5b34-4f03-bdf3-99d8e6aa830c" Oct 30 13:00:23.126410 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:23.126380 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv"] Oct 30 13:00:23.513234 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:23.513197 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42de5f66-15b6-4383-801e-963988e0ecc6" path="/var/lib/kubelet/pods/42de5f66-15b6-4383-801e-963988e0ecc6/volumes" Oct 30 13:00:23.513679 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:23.513662 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="995b547e-ad4b-46a1-b82c-f29212e99505" path="/var/lib/kubelet/pods/995b547e-ad4b-46a1-b82c-f29212e99505/volumes" Oct 30 13:00:24.206408 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:24.206380 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv" Oct 30 13:00:24.344249 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:24.344176 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58fed038-5b34-4f03-bdf3-99d8e6aa830c-kube-api-access-fhcgn\" (UniqueName: \"kubernetes.io/projected/58fed038-5b34-4f03-bdf3-99d8e6aa830c-kube-api-access-fhcgn\") pod \"58fed038-5b34-4f03-bdf3-99d8e6aa830c\" (UID: \"58fed038-5b34-4f03-bdf3-99d8e6aa830c\") " Oct 30 13:00:24.344249 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:24.344211 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58fed038-5b34-4f03-bdf3-99d8e6aa830c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58fed038-5b34-4f03-bdf3-99d8e6aa830c-maas-api-service-ca\") pod \"58fed038-5b34-4f03-bdf3-99d8e6aa830c\" (UID: \"58fed038-5b34-4f03-bdf3-99d8e6aa830c\") " Oct 30 13:00:24.344577 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:24.344556 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58fed038-5b34-4f03-bdf3-99d8e6aa830c-maas-api-service-ca" pod "58fed038-5b34-4f03-bdf3-99d8e6aa830c" (UID: "58fed038-5b34-4f03-bdf3-99d8e6aa830c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:24.346238 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:24.346213 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58fed038-5b34-4f03-bdf3-99d8e6aa830c-kube-api-access-fhcgn" pod "58fed038-5b34-4f03-bdf3-99d8e6aa830c" (UID: "58fed038-5b34-4f03-bdf3-99d8e6aa830c"). InnerVolumeSpecName "kube-api-access-fhcgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:24.445293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:24.445229 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhcgn\" (UniqueName: \"kubernetes.io/projected/58fed038-5b34-4f03-bdf3-99d8e6aa830c-kube-api-access-fhcgn\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:00:24.445293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:24.445260 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58fed038-5b34-4f03-bdf3-99d8e6aa830c-maas-api-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:00:25.067078 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:25.067044 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv" Oct 30 13:00:25.067078 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:25.067068 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv" event={"ID":"58fed038-5b34-4f03-bdf3-99d8e6aa830c","Type":"ContainerDied","Data":"94d75278435c7c6ee18941b99483a4d2c1f8950a5ad27420c429f810e49e46d3"} Oct 30 13:00:25.067499 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:25.067132 4931 scope.go:122] "RemoveContainer" containerID="5e70d27b0367c205b27c9f432bcbfc6c116787cd7c9ce8999d2616521260e0bb" Oct 30 13:00:25.123491 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:25.123459 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv"] Oct 30 13:00:25.136449 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:25.136425 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889420-xgvxv"] Oct 30 13:00:25.513084 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:25.513049 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58fed038-5b34-4f03-bdf3-99d8e6aa830c" path="/var/lib/kubelet/pods/58fed038-5b34-4f03-bdf3-99d8e6aa830c/volumes" Oct 30 13:00:30.389952 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.389901 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk"] Oct 30 13:00:30.390478 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.390461 4931 cpu_manager.go:414] "containerMap: removing container" podUID="42de5f66-15b6-4383-801e-963988e0ecc6" containerName="cleanup" Oct 30 13:00:30.390525 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.390480 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42de5f66-15b6-4383-801e-963988e0ecc6" containerName="cleanup" Oct 30 13:00:30.390525 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.390491 4931 cpu_manager.go:414] "containerMap: removing container" podUID="4bd13191-96b6-414b-be8c-91cb2cc46035" containerName="cleanup" Oct 30 13:00:30.390525 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.390496 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bd13191-96b6-414b-be8c-91cb2cc46035" containerName="cleanup" Oct 30 13:00:30.390525 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.390511 4931 cpu_manager.go:414] "containerMap: removing container" podUID="58fed038-5b34-4f03-bdf3-99d8e6aa830c" containerName="cleanup" Oct 30 13:00:30.390525 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.390517 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58fed038-5b34-4f03-bdf3-99d8e6aa830c" containerName="cleanup" Oct 30 13:00:30.390525 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.390525 4931 cpu_manager.go:414] "containerMap: removing container" podUID="995b547e-ad4b-46a1-b82c-f29212e99505" containerName="cleanup" Oct 30 13:00:30.390830 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.390531 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="995b547e-ad4b-46a1-b82c-f29212e99505" containerName="cleanup" Oct 30 13:00:30.390830 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.390537 4931 cpu_manager.go:414] "containerMap: removing container" podUID="995b547e-ad4b-46a1-b82c-f29212e99505" containerName="cleanup" Oct 30 13:00:30.390830 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.390541 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="995b547e-ad4b-46a1-b82c-f29212e99505" containerName="cleanup" Oct 30 13:00:30.390830 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.390551 4931 cpu_manager.go:414] "containerMap: removing container" podUID="58fed038-5b34-4f03-bdf3-99d8e6aa830c" containerName="cleanup" Oct 30 13:00:30.390830 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.390555 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58fed038-5b34-4f03-bdf3-99d8e6aa830c" containerName="cleanup" Oct 30 13:00:30.390830 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.390569 4931 cpu_manager.go:414] "containerMap: removing container" podUID="42de5f66-15b6-4383-801e-963988e0ecc6" containerName="cleanup" Oct 30 13:00:30.390830 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.390573 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42de5f66-15b6-4383-801e-963988e0ecc6" containerName="cleanup" Oct 30 13:00:30.390830 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.390589 4931 cpu_manager.go:414] "containerMap: removing container" podUID="4bd13191-96b6-414b-be8c-91cb2cc46035" containerName="cleanup" Oct 30 13:00:30.390830 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.390594 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bd13191-96b6-414b-be8c-91cb2cc46035" containerName="cleanup" Oct 30 13:00:30.390830 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.390602 4931 cpu_manager.go:414] "containerMap: removing container" podUID="4bd13191-96b6-414b-be8c-91cb2cc46035" containerName="cleanup" Oct 30 13:00:30.390830 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.390606 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bd13191-96b6-414b-be8c-91cb2cc46035" containerName="cleanup" Oct 30 13:00:30.390830 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.390684 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="42de5f66-15b6-4383-801e-963988e0ecc6" containerName="cleanup" Oct 30 13:00:30.390830 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.390692 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bd13191-96b6-414b-be8c-91cb2cc46035" containerName="cleanup" Oct 30 13:00:30.390830 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.390698 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="58fed038-5b34-4f03-bdf3-99d8e6aa830c" containerName="cleanup" Oct 30 13:00:30.390830 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.390703 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="995b547e-ad4b-46a1-b82c-f29212e99505" containerName="cleanup" Oct 30 13:00:30.390830 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.390708 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="995b547e-ad4b-46a1-b82c-f29212e99505" containerName="cleanup" Oct 30 13:00:30.390830 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.390714 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="58fed038-5b34-4f03-bdf3-99d8e6aa830c" containerName="cleanup" Oct 30 13:00:30.390830 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.390722 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bd13191-96b6-414b-be8c-91cb2cc46035" containerName="cleanup" Oct 30 13:00:30.390830 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.390726 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="42de5f66-15b6-4383-801e-963988e0ecc6" containerName="cleanup" Oct 30 13:00:30.390830 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.390732 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="58fed038-5b34-4f03-bdf3-99d8e6aa830c" containerName="cleanup" Oct 30 13:00:30.397389 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.397363 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:00:30.427584 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.427549 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-8b46c9\"/\"premium-w1-8b46c9-kserve-self-signed-certs\"" Oct 30 13:00:30.430259 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.430234 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk"] Oct 30 13:00:30.509523 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.509485 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9577f571-30cd-4043-a3e4-72bf95837552-home\") pod \"premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk\" (UID: \"9577f571-30cd-4043-a3e4-72bf95837552\") " pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:00:30.509767 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.509534 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9577f571-30cd-4043-a3e4-72bf95837552-tmp-dir\") pod \"premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk\" (UID: \"9577f571-30cd-4043-a3e4-72bf95837552\") " pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:00:30.509767 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.509717 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9577f571-30cd-4043-a3e4-72bf95837552-tls-certs\") pod \"premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk\" (UID: \"9577f571-30cd-4043-a3e4-72bf95837552\") " pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:00:30.509884 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.509809 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9577f571-30cd-4043-a3e4-72bf95837552-model-cache\") pod \"premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk\" (UID: \"9577f571-30cd-4043-a3e4-72bf95837552\") " pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:00:30.509884 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.509840 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dtgp\" (UniqueName: \"kubernetes.io/projected/9577f571-30cd-4043-a3e4-72bf95837552-kube-api-access-7dtgp\") pod \"premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk\" (UID: \"9577f571-30cd-4043-a3e4-72bf95837552\") " pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:00:30.509985 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.509895 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9577f571-30cd-4043-a3e4-72bf95837552-dshm\") pod \"premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk\" (UID: \"9577f571-30cd-4043-a3e4-72bf95837552\") " pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:00:30.611134 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.611083 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9577f571-30cd-4043-a3e4-72bf95837552-tls-certs\") pod \"premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk\" (UID: \"9577f571-30cd-4043-a3e4-72bf95837552\") " pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:00:30.611351 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.611248 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9577f571-30cd-4043-a3e4-72bf95837552-model-cache\") pod \"premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk\" (UID: \"9577f571-30cd-4043-a3e4-72bf95837552\") " pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:00:30.611351 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.611302 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dtgp\" (UniqueName: \"kubernetes.io/projected/9577f571-30cd-4043-a3e4-72bf95837552-kube-api-access-7dtgp\") pod \"premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk\" (UID: \"9577f571-30cd-4043-a3e4-72bf95837552\") " pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:00:30.611351 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.611329 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9577f571-30cd-4043-a3e4-72bf95837552-dshm\") pod \"premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk\" (UID: \"9577f571-30cd-4043-a3e4-72bf95837552\") " pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:00:30.611538 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.611398 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9577f571-30cd-4043-a3e4-72bf95837552-home\") pod \"premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk\" (UID: \"9577f571-30cd-4043-a3e4-72bf95837552\") " pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:00:30.611538 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.611425 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9577f571-30cd-4043-a3e4-72bf95837552-tmp-dir\") pod \"premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk\" (UID: \"9577f571-30cd-4043-a3e4-72bf95837552\") " pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:00:30.611922 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.611891 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9577f571-30cd-4043-a3e4-72bf95837552-home\") pod \"premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk\" (UID: \"9577f571-30cd-4043-a3e4-72bf95837552\") " pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:00:30.612109 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.611943 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9577f571-30cd-4043-a3e4-72bf95837552-tmp-dir\") pod \"premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk\" (UID: \"9577f571-30cd-4043-a3e4-72bf95837552\") " pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:00:30.612109 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.612062 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9577f571-30cd-4043-a3e4-72bf95837552-model-cache\") pod \"premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk\" (UID: \"9577f571-30cd-4043-a3e4-72bf95837552\") " pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:00:30.614175 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.614148 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9577f571-30cd-4043-a3e4-72bf95837552-dshm\") pod \"premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk\" (UID: \"9577f571-30cd-4043-a3e4-72bf95837552\") " pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:00:30.614278 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.614187 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9577f571-30cd-4043-a3e4-72bf95837552-tls-certs\") pod \"premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk\" (UID: \"9577f571-30cd-4043-a3e4-72bf95837552\") " pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:00:30.636302 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.636277 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dtgp\" (UniqueName: \"kubernetes.io/projected/9577f571-30cd-4043-a3e4-72bf95837552-kube-api-access-7dtgp\") pod \"premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk\" (UID: \"9577f571-30cd-4043-a3e4-72bf95837552\") " pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:00:30.710280 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:30.710244 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:00:31.018956 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:31.018843 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk"] Oct 30 13:00:31.021327 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:00:31.021302 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9577f571_30cd_4043_a3e4_72bf95837552.slice/crio-4184fd36f69f0deda36442fb4c1a16cfc2be62b79f7b983e63e5ecb70ec0f2b2 WatchSource:0}: Error finding container 4184fd36f69f0deda36442fb4c1a16cfc2be62b79f7b983e63e5ecb70ec0f2b2: Status 404 returned error can't find the container with id 4184fd36f69f0deda36442fb4c1a16cfc2be62b79f7b983e63e5ecb70ec0f2b2 Oct 30 13:00:31.108556 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:31.108503 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" event={"ID":"9577f571-30cd-4043-a3e4-72bf95837552","Type":"ContainerStarted","Data":"c3ef6595450b537b0e3c08895c3069621cb7d582458f5ca47f4232bc2d60d04f"} Oct 30 13:00:31.108723 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:31.108564 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" event={"ID":"9577f571-30cd-4043-a3e4-72bf95837552","Type":"ContainerStarted","Data":"4184fd36f69f0deda36442fb4c1a16cfc2be62b79f7b983e63e5ecb70ec0f2b2"} Oct 30 13:00:31.165519 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:31.165463 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" podStartSLOduration=1.165450126 podStartE2EDuration="1.165450126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:31.163171832 +0000 UTC m=+929.766670278" watchObservedRunningTime="2026-10-30 13:00:31.165450126 +0000 UTC m=+929.768948622" Oct 30 13:00:31.711338 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:31.711307 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:00:33.674647 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:33.674605 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-775a7fbd-68rv9"] Oct 30 13:00:33.675112 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:33.675097 4931 cpu_manager.go:414] "containerMap: removing container" podUID="58fed038-5b34-4f03-bdf3-99d8e6aa830c" containerName="cleanup" Oct 30 13:00:33.675165 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:33.675114 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58fed038-5b34-4f03-bdf3-99d8e6aa830c" containerName="cleanup" Oct 30 13:00:33.675165 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:33.675125 4931 cpu_manager.go:414] "containerMap: removing container" podUID="995b547e-ad4b-46a1-b82c-f29212e99505" containerName="cleanup" Oct 30 13:00:33.675165 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:33.675130 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="995b547e-ad4b-46a1-b82c-f29212e99505" containerName="cleanup" Oct 30 13:00:33.675365 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:33.675175 4931 cpu_manager.go:414] "containerMap: removing container" podUID="42de5f66-15b6-4383-801e-963988e0ecc6" containerName="cleanup" Oct 30 13:00:33.675365 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:33.675181 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42de5f66-15b6-4383-801e-963988e0ecc6" containerName="cleanup" Oct 30 13:00:33.675365 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:33.675261 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="42de5f66-15b6-4383-801e-963988e0ecc6" containerName="cleanup" Oct 30 13:00:33.675365 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:33.675282 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bd13191-96b6-414b-be8c-91cb2cc46035" containerName="cleanup" Oct 30 13:00:33.675365 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:33.675290 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="995b547e-ad4b-46a1-b82c-f29212e99505" containerName="cleanup" Oct 30 13:00:33.682182 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:33.682163 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775a7fbd-68rv9" Oct 30 13:00:33.699229 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:33.699200 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-775a7fbd-68rv9"] Oct 30 13:00:33.709302 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:33.709258 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:00:33.709454 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:33.709301 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:00:33.739182 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:33.739151 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl2jr\" (UniqueName: \"kubernetes.io/projected/940c5766-5edb-44af-b989-7fe49f3b8e99-kube-api-access-dl2jr\") pod \"maas-api-revoke-sub-775a7fbd-68rv9\" (UID: \"940c5766-5edb-44af-b989-7fe49f3b8e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775a7fbd-68rv9" Oct 30 13:00:33.739371 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:33.739223 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/940c5766-5edb-44af-b989-7fe49f3b8e99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-775a7fbd-68rv9\" (UID: \"940c5766-5edb-44af-b989-7fe49f3b8e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775a7fbd-68rv9" Oct 30 13:00:33.739371 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:33.739335 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/940c5766-5edb-44af-b989-7fe49f3b8e99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-775a7fbd-68rv9\" (UID: \"940c5766-5edb-44af-b989-7fe49f3b8e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775a7fbd-68rv9" Oct 30 13:00:33.840931 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:33.840896 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/940c5766-5edb-44af-b989-7fe49f3b8e99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-775a7fbd-68rv9\" (UID: \"940c5766-5edb-44af-b989-7fe49f3b8e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775a7fbd-68rv9" Oct 30 13:00:33.841141 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:33.840987 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl2jr\" (UniqueName: \"kubernetes.io/projected/940c5766-5edb-44af-b989-7fe49f3b8e99-kube-api-access-dl2jr\") pod \"maas-api-revoke-sub-775a7fbd-68rv9\" (UID: \"940c5766-5edb-44af-b989-7fe49f3b8e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775a7fbd-68rv9" Oct 30 13:00:33.841141 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:33.841032 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/940c5766-5edb-44af-b989-7fe49f3b8e99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-775a7fbd-68rv9\" (UID: \"940c5766-5edb-44af-b989-7fe49f3b8e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775a7fbd-68rv9" Oct 30 13:00:33.841614 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:33.841591 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/940c5766-5edb-44af-b989-7fe49f3b8e99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-775a7fbd-68rv9\" (UID: \"940c5766-5edb-44af-b989-7fe49f3b8e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775a7fbd-68rv9" Oct 30 13:00:33.843312 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:33.843294 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/940c5766-5edb-44af-b989-7fe49f3b8e99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-775a7fbd-68rv9\" (UID: \"940c5766-5edb-44af-b989-7fe49f3b8e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775a7fbd-68rv9" Oct 30 13:00:33.865527 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:33.865491 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl2jr\" (UniqueName: \"kubernetes.io/projected/940c5766-5edb-44af-b989-7fe49f3b8e99-kube-api-access-dl2jr\") pod \"maas-api-revoke-sub-775a7fbd-68rv9\" (UID: \"940c5766-5edb-44af-b989-7fe49f3b8e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775a7fbd-68rv9" Oct 30 13:00:33.997169 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:33.997135 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775a7fbd-68rv9" Oct 30 13:00:34.310907 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:34.310875 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-775a7fbd-68rv9"] Oct 30 13:00:34.312312 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:00:34.312282 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c5766_5edb_44af_b989_7fe49f3b8e99.slice/crio-0adf34324a6c3daaff62b291b89d0934af57b2d1de797a4298ea0a14a86fd1d5 WatchSource:0}: Error finding container 0adf34324a6c3daaff62b291b89d0934af57b2d1de797a4298ea0a14a86fd1d5: Status 404 returned error can't find the container with id 0adf34324a6c3daaff62b291b89d0934af57b2d1de797a4298ea0a14a86fd1d5 Oct 30 13:00:35.136919 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:35.136880 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775a7fbd-68rv9" event={"ID":"940c5766-5edb-44af-b989-7fe49f3b8e99","Type":"ContainerStarted","Data":"7648a74916883ae5d2e60c855a47e4066871dd0291f88ae2c1490c31afc4bba7"} Oct 30 13:00:35.137345 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:35.136964 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775a7fbd-68rv9" event={"ID":"940c5766-5edb-44af-b989-7fe49f3b8e99","Type":"ContainerStarted","Data":"0adf34324a6c3daaff62b291b89d0934af57b2d1de797a4298ea0a14a86fd1d5"} Oct 30 13:00:35.180939 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:35.180892 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775a7fbd-68rv9" podStartSLOduration=2.180878524 podStartE2EDuration="2.180878524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:35.179462897 +0000 UTC m=+933.782961343" watchObservedRunningTime="2026-10-30 13:00:35.180878524 +0000 UTC m=+933.784376970" Oct 30 13:00:36.143953 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:36.143920 4931 generic.go:356] "Generic (PLEG): container finished" podID="940c5766-5edb-44af-b989-7fe49f3b8e99" containerID="7648a74916883ae5d2e60c855a47e4066871dd0291f88ae2c1490c31afc4bba7" exitCode=0 Oct 30 13:00:36.144386 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:36.144010 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775a7fbd-68rv9" event={"ID":"940c5766-5edb-44af-b989-7fe49f3b8e99","Type":"ContainerDied","Data":"7648a74916883ae5d2e60c855a47e4066871dd0291f88ae2c1490c31afc4bba7"} Oct 30 13:00:37.289217 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:37.289188 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775a7fbd-68rv9" Oct 30 13:00:37.355771 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:00:37.355736 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:00:37.377359 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:37.377329 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/940c5766-5edb-44af-b989-7fe49f3b8e99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/940c5766-5edb-44af-b989-7fe49f3b8e99-maas-api-xks-ca\") pod \"940c5766-5edb-44af-b989-7fe49f3b8e99\" (UID: \"940c5766-5edb-44af-b989-7fe49f3b8e99\") " Oct 30 13:00:37.377504 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:37.377438 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/940c5766-5edb-44af-b989-7fe49f3b8e99-kube-api-access-dl2jr\" (UniqueName: \"kubernetes.io/projected/940c5766-5edb-44af-b989-7fe49f3b8e99-kube-api-access-dl2jr\") pod \"940c5766-5edb-44af-b989-7fe49f3b8e99\" (UID: \"940c5766-5edb-44af-b989-7fe49f3b8e99\") " Oct 30 13:00:37.377504 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:37.377495 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/940c5766-5edb-44af-b989-7fe49f3b8e99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/940c5766-5edb-44af-b989-7fe49f3b8e99-maas-api-openshift-ca\") pod \"940c5766-5edb-44af-b989-7fe49f3b8e99\" (UID: \"940c5766-5edb-44af-b989-7fe49f3b8e99\") " Oct 30 13:00:37.377899 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:37.377868 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/940c5766-5edb-44af-b989-7fe49f3b8e99-maas-api-openshift-ca" pod "940c5766-5edb-44af-b989-7fe49f3b8e99" (UID: "940c5766-5edb-44af-b989-7fe49f3b8e99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:37.379443 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:37.379420 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/940c5766-5edb-44af-b989-7fe49f3b8e99-maas-api-xks-ca" pod "940c5766-5edb-44af-b989-7fe49f3b8e99" (UID: "940c5766-5edb-44af-b989-7fe49f3b8e99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:00:37.379622 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:37.379595 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/940c5766-5edb-44af-b989-7fe49f3b8e99-kube-api-access-dl2jr" pod "940c5766-5edb-44af-b989-7fe49f3b8e99" (UID: "940c5766-5edb-44af-b989-7fe49f3b8e99"). InnerVolumeSpecName "kube-api-access-dl2jr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:37.478627 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:37.478592 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/940c5766-5edb-44af-b989-7fe49f3b8e99-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:00:37.478627 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:37.478625 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl2jr\" (UniqueName: \"kubernetes.io/projected/940c5766-5edb-44af-b989-7fe49f3b8e99-kube-api-access-dl2jr\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:00:37.478627 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:37.478636 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/940c5766-5edb-44af-b989-7fe49f3b8e99-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:00:38.156337 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:38.156309 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775a7fbd-68rv9" Oct 30 13:00:38.156509 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:38.156312 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775a7fbd-68rv9" event={"ID":"940c5766-5edb-44af-b989-7fe49f3b8e99","Type":"ContainerDied","Data":"0adf34324a6c3daaff62b291b89d0934af57b2d1de797a4298ea0a14a86fd1d5"} Oct 30 13:00:38.156509 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:38.156444 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0adf34324a6c3daaff62b291b89d0934af57b2d1de797a4298ea0a14a86fd1d5" Oct 30 13:00:40.135247 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.135205 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h"] Oct 30 13:00:40.135920 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.135859 4931 cpu_manager.go:414] "containerMap: removing container" podUID="940c5766-5edb-44af-b989-7fe49f3b8e99" containerName="revoke-keys" Oct 30 13:00:40.135920 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.135882 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="940c5766-5edb-44af-b989-7fe49f3b8e99" containerName="revoke-keys" Oct 30 13:00:40.136081 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.136071 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="940c5766-5edb-44af-b989-7fe49f3b8e99" containerName="revoke-keys" Oct 30 13:00:40.139774 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.139751 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" Oct 30 13:00:40.157151 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.157123 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-c90b9e-openshift-default-dockercfg-6pcbf\"" Oct 30 13:00:40.173512 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.173481 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h"] Oct 30 13:00:40.209495 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.209463 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bw9s\" (UniqueName: \"kubernetes.io/projected/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-kube-api-access-9bw9s\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" Oct 30 13:00:40.209672 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.209504 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-istiod-ca-cert\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" Oct 30 13:00:40.209672 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.209583 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-workload-certs\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" Oct 30 13:00:40.209672 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.209604 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-credential-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" Oct 30 13:00:40.209672 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.209623 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-istio-token\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" Oct 30 13:00:40.209672 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.209652 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-istio-envoy\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" Oct 30 13:00:40.209847 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.209765 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-workload-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" Oct 30 13:00:40.209847 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.209837 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-istio-data\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" Oct 30 13:00:40.209910 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.209870 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-istio-podinfo\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" Oct 30 13:00:40.236617 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.236581 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:00:40.311430 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.311394 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-workload-certs\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" Oct 30 13:00:40.311430 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.311432 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-credential-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" Oct 30 13:00:40.311679 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.311454 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-istio-token\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" Oct 30 13:00:40.311679 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.311505 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-istio-envoy\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" Oct 30 13:00:40.311679 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.311553 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-workload-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" Oct 30 13:00:40.311679 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.311668 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-istio-data\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" Oct 30 13:00:40.311889 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.311717 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-istio-podinfo\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" Oct 30 13:00:40.311889 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.311804 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bw9s\" (UniqueName: \"kubernetes.io/projected/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-kube-api-access-9bw9s\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" Oct 30 13:00:40.311889 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.311847 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-istiod-ca-cert\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" Oct 30 13:00:40.312053 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.311935 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-credential-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" Oct 30 13:00:40.312053 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.312000 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-istio-data\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" Oct 30 13:00:40.312158 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.312098 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-workload-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" Oct 30 13:00:40.312390 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.312365 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-workload-certs\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" Oct 30 13:00:40.312632 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.312563 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-istiod-ca-cert\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" Oct 30 13:00:40.314417 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.314386 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-istio-envoy\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" Oct 30 13:00:40.314790 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.314769 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-istio-podinfo\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" Oct 30 13:00:40.338940 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.338907 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-istio-token\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" Oct 30 13:00:40.339104 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.339082 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bw9s\" (UniqueName: \"kubernetes.io/projected/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-kube-api-access-9bw9s\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" Oct 30 13:00:40.453984 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.453948 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" Oct 30 13:00:40.710992 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.710905 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:00:40.724004 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.723974 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:00:40.780057 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.780013 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h"] Oct 30 13:00:40.787080 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:00:40.787044 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcbc4d0a_6516_4cc5_ab8a_0e19a36e3ae9.slice/crio-9db080bda909a79e21a4da49a194e9fd0e8a3782ef05e060e0f25733ddf6e046 WatchSource:0}: Error finding container 9db080bda909a79e21a4da49a194e9fd0e8a3782ef05e060e0f25733ddf6e046: Status 404 returned error can't find the container with id 9db080bda909a79e21a4da49a194e9fd0e8a3782ef05e060e0f25733ddf6e046 Oct 30 13:00:40.789823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.789786 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:00:40.789951 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.789853 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:00:40.789951 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:40.789895 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:00:41.178027 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:41.177909 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" event={"ID":"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9","Type":"ContainerStarted","Data":"f72beba7d76820188fe8ca1245ec15c65c499707d539fe2d36631615633b6fb0"} Oct 30 13:00:41.178027 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:41.177985 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" event={"ID":"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9","Type":"ContainerStarted","Data":"9db080bda909a79e21a4da49a194e9fd0e8a3782ef05e060e0f25733ddf6e046"} Oct 30 13:00:41.189462 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:41.189438 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:00:41.269689 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:41.269635 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" podStartSLOduration=1.269619249 podStartE2EDuration="1.269619249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:41.266731269 +0000 UTC m=+939.870229734" watchObservedRunningTime="2026-10-30 13:00:41.269619249 +0000 UTC m=+939.873117692" Oct 30 13:00:41.458289 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:41.455638 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" Oct 30 13:00:41.458636 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:41.458609 4931 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 13:00:41.458854 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:41.458812 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" podUID="fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 13:00:42.457740 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:42.457682 4931 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 13:00:42.458198 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:42.457773 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" podUID="fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 13:00:43.457650 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:43.457611 4931 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 13:00:43.457825 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:43.457680 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" podUID="fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 13:00:44.457706 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:44.457667 4931 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 13:00:44.457901 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:44.457731 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" podUID="fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 13:00:45.455631 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:45.455595 4931 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 13:00:45.455905 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:45.455878 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" podUID="fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 13:00:46.457632 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:46.457591 4931 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 13:00:46.458410 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:46.457669 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" podUID="fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 13:00:47.514487 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:47.514452 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" Oct 30 13:00:47.515253 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:47.515227 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" Oct 30 13:00:47.516015 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:47.515996 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" Oct 30 13:00:49.074634 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:49.074602 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac033438-rhwjd"] Oct 30 13:00:49.078281 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:49.078244 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac033438-rhwjd" Oct 30 13:00:49.094704 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:49.094684 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:00:49.107998 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:49.107975 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:00:49.110656 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:49.110632 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac033438-rhwjd"] Oct 30 13:00:49.129990 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:49.129961 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqrpb\" (UniqueName: \"kubernetes.io/projected/06e41c6f-53ac-4719-b4f2-7631d6acbbef-kube-api-access-bqrpb\") pod \"maas-api-revoke-sub-ac033438-rhwjd\" (UID: \"06e41c6f-53ac-4719-b4f2-7631d6acbbef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac033438-rhwjd" Oct 30 13:00:49.130141 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:49.130045 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e41c6f-53ac-4719-b4f2-7631d6acbbef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac033438-rhwjd\" (UID: \"06e41c6f-53ac-4719-b4f2-7631d6acbbef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac033438-rhwjd" Oct 30 13:00:49.130185 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:49.130146 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e41c6f-53ac-4719-b4f2-7631d6acbbef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac033438-rhwjd\" (UID: \"06e41c6f-53ac-4719-b4f2-7631d6acbbef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac033438-rhwjd" Oct 30 13:00:49.231339 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:49.231311 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqrpb\" (UniqueName: \"kubernetes.io/projected/06e41c6f-53ac-4719-b4f2-7631d6acbbef-kube-api-access-bqrpb\") pod \"maas-api-revoke-sub-ac033438-rhwjd\" (UID: \"06e41c6f-53ac-4719-b4f2-7631d6acbbef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac033438-rhwjd" Oct 30 13:00:49.231477 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:49.231366 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e41c6f-53ac-4719-b4f2-7631d6acbbef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac033438-rhwjd\" (UID: \"06e41c6f-53ac-4719-b4f2-7631d6acbbef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac033438-rhwjd" Oct 30 13:00:49.231477 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:49.231393 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e41c6f-53ac-4719-b4f2-7631d6acbbef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac033438-rhwjd\" (UID: \"06e41c6f-53ac-4719-b4f2-7631d6acbbef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac033438-rhwjd" Oct 30 13:00:49.232048 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:49.232025 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e41c6f-53ac-4719-b4f2-7631d6acbbef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac033438-rhwjd\" (UID: \"06e41c6f-53ac-4719-b4f2-7631d6acbbef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac033438-rhwjd" Oct 30 13:00:49.233643 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:49.233622 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e41c6f-53ac-4719-b4f2-7631d6acbbef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac033438-rhwjd\" (UID: \"06e41c6f-53ac-4719-b4f2-7631d6acbbef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac033438-rhwjd" Oct 30 13:00:49.255460 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:49.255441 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqrpb\" (UniqueName: \"kubernetes.io/projected/06e41c6f-53ac-4719-b4f2-7631d6acbbef-kube-api-access-bqrpb\") pod \"maas-api-revoke-sub-ac033438-rhwjd\" (UID: \"06e41c6f-53ac-4719-b4f2-7631d6acbbef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac033438-rhwjd" Oct 30 13:00:49.391584 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:49.391495 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac033438-rhwjd" Oct 30 13:00:49.691380 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:49.691353 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac033438-rhwjd"] Oct 30 13:00:49.693588 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:00:49.693558 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e41c6f_53ac_4719_b4f2_7631d6acbbef.slice/crio-d3417a197254c5955ac7234232b48b861b8188c87c7dcdddd625f9573b36eaa2 WatchSource:0}: Error finding container d3417a197254c5955ac7234232b48b861b8188c87c7dcdddd625f9573b36eaa2: Status 404 returned error can't find the container with id d3417a197254c5955ac7234232b48b861b8188c87c7dcdddd625f9573b36eaa2 Oct 30 13:00:50.237539 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:50.237458 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac033438-rhwjd" event={"ID":"06e41c6f-53ac-4719-b4f2-7631d6acbbef","Type":"ContainerStarted","Data":"c4e976747dc6fd8ee8ba3a1bf81137e440610c84df416b09b66618b109fb0d04"} Oct 30 13:00:50.237539 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:50.237512 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac033438-rhwjd" event={"ID":"06e41c6f-53ac-4719-b4f2-7631d6acbbef","Type":"ContainerStarted","Data":"d3417a197254c5955ac7234232b48b861b8188c87c7dcdddd625f9573b36eaa2"} Oct 30 13:00:50.297579 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:50.297527 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac033438-rhwjd" podStartSLOduration=1.297514762 podStartE2EDuration="1.297514762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:50.283860573 +0000 UTC m=+948.887359019" watchObservedRunningTime="2026-10-30 13:00:50.297514762 +0000 UTC m=+948.901013207" Oct 30 13:00:51.244580 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:51.244545 4931 generic.go:356] "Generic (PLEG): container finished" podID="06e41c6f-53ac-4719-b4f2-7631d6acbbef" containerID="c4e976747dc6fd8ee8ba3a1bf81137e440610c84df416b09b66618b109fb0d04" exitCode=0 Oct 30 13:00:51.244994 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:51.244627 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac033438-rhwjd" event={"ID":"06e41c6f-53ac-4719-b4f2-7631d6acbbef","Type":"ContainerDied","Data":"c4e976747dc6fd8ee8ba3a1bf81137e440610c84df416b09b66618b109fb0d04"} Oct 30 13:00:52.565299 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:52.562308 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac033438-rhwjd" Oct 30 13:00:52.686096 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:52.686065 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06e41c6f-53ac-4719-b4f2-7631d6acbbef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e41c6f-53ac-4719-b4f2-7631d6acbbef-maas-api-openshift-ca\") pod \"06e41c6f-53ac-4719-b4f2-7631d6acbbef\" (UID: \"06e41c6f-53ac-4719-b4f2-7631d6acbbef\") " Oct 30 13:00:52.686281 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:52.686256 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06e41c6f-53ac-4719-b4f2-7631d6acbbef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e41c6f-53ac-4719-b4f2-7631d6acbbef-maas-api-xks-ca\") pod \"06e41c6f-53ac-4719-b4f2-7631d6acbbef\" (UID: \"06e41c6f-53ac-4719-b4f2-7631d6acbbef\") " Oct 30 13:00:52.686398 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:52.686387 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06e41c6f-53ac-4719-b4f2-7631d6acbbef-kube-api-access-bqrpb\" (UniqueName: \"kubernetes.io/projected/06e41c6f-53ac-4719-b4f2-7631d6acbbef-kube-api-access-bqrpb\") pod \"06e41c6f-53ac-4719-b4f2-7631d6acbbef\" (UID: \"06e41c6f-53ac-4719-b4f2-7631d6acbbef\") " Oct 30 13:00:52.687441 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:52.687421 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06e41c6f-53ac-4719-b4f2-7631d6acbbef-maas-api-openshift-ca" pod "06e41c6f-53ac-4719-b4f2-7631d6acbbef" (UID: "06e41c6f-53ac-4719-b4f2-7631d6acbbef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:52.691371 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:52.691339 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e41c6f-53ac-4719-b4f2-7631d6acbbef-kube-api-access-bqrpb" pod "06e41c6f-53ac-4719-b4f2-7631d6acbbef" (UID: "06e41c6f-53ac-4719-b4f2-7631d6acbbef"). InnerVolumeSpecName "kube-api-access-bqrpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:52.693102 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:52.693065 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06e41c6f-53ac-4719-b4f2-7631d6acbbef-maas-api-xks-ca" pod "06e41c6f-53ac-4719-b4f2-7631d6acbbef" (UID: "06e41c6f-53ac-4719-b4f2-7631d6acbbef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:00:52.787933 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:52.787838 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e41c6f-53ac-4719-b4f2-7631d6acbbef-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:00:52.787933 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:52.787882 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e41c6f-53ac-4719-b4f2-7631d6acbbef-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:00:52.787933 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:52.787898 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqrpb\" (UniqueName: \"kubernetes.io/projected/06e41c6f-53ac-4719-b4f2-7631d6acbbef-kube-api-access-bqrpb\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:00:53.274290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:53.273558 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac033438-rhwjd" event={"ID":"06e41c6f-53ac-4719-b4f2-7631d6acbbef","Type":"ContainerDied","Data":"d3417a197254c5955ac7234232b48b861b8188c87c7dcdddd625f9573b36eaa2"} Oct 30 13:00:53.274290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:53.273625 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3417a197254c5955ac7234232b48b861b8188c87c7dcdddd625f9573b36eaa2" Oct 30 13:00:53.274290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:00:53.273718 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac033438-rhwjd" Oct 30 13:01:00.702727 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:00.702693 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-479d95c1-thznf"] Oct 30 13:01:00.703404 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:00.703388 4931 cpu_manager.go:414] "containerMap: removing container" podUID="06e41c6f-53ac-4719-b4f2-7631d6acbbef" containerName="revoke-keys" Oct 30 13:01:00.703462 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:00.703406 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06e41c6f-53ac-4719-b4f2-7631d6acbbef" containerName="revoke-keys" Oct 30 13:01:00.703536 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:00.703525 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="06e41c6f-53ac-4719-b4f2-7631d6acbbef" containerName="revoke-keys" Oct 30 13:01:00.707454 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:00.707423 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-479d95c1-thznf" Oct 30 13:01:00.725240 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:00.725207 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-479d95c1-thznf"] Oct 30 13:01:00.735422 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:00.735392 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:01:00.735613 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:00.735414 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:01:00.885491 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:00.885460 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4dpw\" (UniqueName: \"kubernetes.io/projected/267c535f-c034-4727-8c4f-55c6565e5ffb-kube-api-access-g4dpw\") pod \"maas-api-revoke-sub-479d95c1-thznf\" (UID: \"267c535f-c034-4727-8c4f-55c6565e5ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-479d95c1-thznf" Oct 30 13:01:00.885664 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:00.885518 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/267c535f-c034-4727-8c4f-55c6565e5ffb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-479d95c1-thznf\" (UID: \"267c535f-c034-4727-8c4f-55c6565e5ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-479d95c1-thznf" Oct 30 13:01:00.885664 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:00.885648 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/267c535f-c034-4727-8c4f-55c6565e5ffb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-479d95c1-thznf\" (UID: \"267c535f-c034-4727-8c4f-55c6565e5ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-479d95c1-thznf" Oct 30 13:01:00.986856 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:00.986771 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/267c535f-c034-4727-8c4f-55c6565e5ffb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-479d95c1-thznf\" (UID: \"267c535f-c034-4727-8c4f-55c6565e5ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-479d95c1-thznf" Oct 30 13:01:00.986999 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:00.986864 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4dpw\" (UniqueName: \"kubernetes.io/projected/267c535f-c034-4727-8c4f-55c6565e5ffb-kube-api-access-g4dpw\") pod \"maas-api-revoke-sub-479d95c1-thznf\" (UID: \"267c535f-c034-4727-8c4f-55c6565e5ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-479d95c1-thznf" Oct 30 13:01:00.986999 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:00.986903 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/267c535f-c034-4727-8c4f-55c6565e5ffb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-479d95c1-thznf\" (UID: \"267c535f-c034-4727-8c4f-55c6565e5ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-479d95c1-thznf" Oct 30 13:01:00.987471 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:00.987449 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/267c535f-c034-4727-8c4f-55c6565e5ffb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-479d95c1-thznf\" (UID: \"267c535f-c034-4727-8c4f-55c6565e5ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-479d95c1-thznf" Oct 30 13:01:00.989303 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:00.989284 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/267c535f-c034-4727-8c4f-55c6565e5ffb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-479d95c1-thznf\" (UID: \"267c535f-c034-4727-8c4f-55c6565e5ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-479d95c1-thznf" Oct 30 13:01:01.011379 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:01.011354 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4dpw\" (UniqueName: \"kubernetes.io/projected/267c535f-c034-4727-8c4f-55c6565e5ffb-kube-api-access-g4dpw\") pod \"maas-api-revoke-sub-479d95c1-thznf\" (UID: \"267c535f-c034-4727-8c4f-55c6565e5ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-479d95c1-thznf" Oct 30 13:01:01.023189 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:01.023165 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-479d95c1-thznf" Oct 30 13:01:01.331112 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:01.331079 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-479d95c1-thznf"] Oct 30 13:01:01.331768 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:01:01.331740 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod267c535f_c034_4727_8c4f_55c6565e5ffb.slice/crio-a73b87856d00e4e556d884d1763f919ac5b8fed7bc50001ffaeaf41142b34aac WatchSource:0}: Error finding container a73b87856d00e4e556d884d1763f919ac5b8fed7bc50001ffaeaf41142b34aac: Status 404 returned error can't find the container with id a73b87856d00e4e556d884d1763f919ac5b8fed7bc50001ffaeaf41142b34aac Oct 30 13:01:01.555683 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:01.555651 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59337106-4z5sd"] Oct 30 13:01:01.560751 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:01.560732 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59337106-4z5sd" Oct 30 13:01:01.579849 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:01.579784 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59337106-4z5sd"] Oct 30 13:01:01.695671 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:01.695635 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s5gs\" (UniqueName: \"kubernetes.io/projected/5d8a2bcf-c3d1-4370-a603-1728b9958a5a-kube-api-access-7s5gs\") pod \"maas-api-revoke-sub-59337106-4z5sd\" (UID: \"5d8a2bcf-c3d1-4370-a603-1728b9958a5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59337106-4z5sd" Oct 30 13:01:01.695671 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:01.695678 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d8a2bcf-c3d1-4370-a603-1728b9958a5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59337106-4z5sd\" (UID: \"5d8a2bcf-c3d1-4370-a603-1728b9958a5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59337106-4z5sd" Oct 30 13:01:01.695885 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:01.695779 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d8a2bcf-c3d1-4370-a603-1728b9958a5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59337106-4z5sd\" (UID: \"5d8a2bcf-c3d1-4370-a603-1728b9958a5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59337106-4z5sd" Oct 30 13:01:01.796375 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:01.796333 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d8a2bcf-c3d1-4370-a603-1728b9958a5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59337106-4z5sd\" (UID: \"5d8a2bcf-c3d1-4370-a603-1728b9958a5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59337106-4z5sd" Oct 30 13:01:01.796857 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:01.796472 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s5gs\" (UniqueName: \"kubernetes.io/projected/5d8a2bcf-c3d1-4370-a603-1728b9958a5a-kube-api-access-7s5gs\") pod \"maas-api-revoke-sub-59337106-4z5sd\" (UID: \"5d8a2bcf-c3d1-4370-a603-1728b9958a5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59337106-4z5sd" Oct 30 13:01:01.796857 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:01.796513 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d8a2bcf-c3d1-4370-a603-1728b9958a5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59337106-4z5sd\" (UID: \"5d8a2bcf-c3d1-4370-a603-1728b9958a5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59337106-4z5sd" Oct 30 13:01:01.797148 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:01.797125 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d8a2bcf-c3d1-4370-a603-1728b9958a5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59337106-4z5sd\" (UID: \"5d8a2bcf-c3d1-4370-a603-1728b9958a5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59337106-4z5sd" Oct 30 13:01:01.798881 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:01.798861 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d8a2bcf-c3d1-4370-a603-1728b9958a5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59337106-4z5sd\" (UID: \"5d8a2bcf-c3d1-4370-a603-1728b9958a5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59337106-4z5sd" Oct 30 13:01:01.821700 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:01.821666 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s5gs\" (UniqueName: \"kubernetes.io/projected/5d8a2bcf-c3d1-4370-a603-1728b9958a5a-kube-api-access-7s5gs\") pod \"maas-api-revoke-sub-59337106-4z5sd\" (UID: \"5d8a2bcf-c3d1-4370-a603-1728b9958a5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59337106-4z5sd" Oct 30 13:01:01.873792 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:01.873714 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59337106-4z5sd" Oct 30 13:01:02.175001 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:02.174975 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59337106-4z5sd"] Oct 30 13:01:02.176852 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:01:02.176823 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d8a2bcf_c3d1_4370_a603_1728b9958a5a.slice/crio-4d91e415e58c9b9895acb3f807befed79de7dd045be240542f8987c0c3967f07 WatchSource:0}: Error finding container 4d91e415e58c9b9895acb3f807befed79de7dd045be240542f8987c0c3967f07: Status 404 returned error can't find the container with id 4d91e415e58c9b9895acb3f807befed79de7dd045be240542f8987c0c3967f07 Oct 30 13:01:02.344339 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:02.344305 4931 generic.go:356] "Generic (PLEG): container finished" podID="267c535f-c034-4727-8c4f-55c6565e5ffb" containerID="457caf1d787225da2e7025663c1ee42ce8b539ae9483a01f258f16a5c507b8f0" exitCode=0 Oct 30 13:01:02.344469 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:02.344390 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-479d95c1-thznf" event={"ID":"267c535f-c034-4727-8c4f-55c6565e5ffb","Type":"ContainerDied","Data":"457caf1d787225da2e7025663c1ee42ce8b539ae9483a01f258f16a5c507b8f0"} Oct 30 13:01:02.344469 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:02.344452 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-479d95c1-thznf" event={"ID":"267c535f-c034-4727-8c4f-55c6565e5ffb","Type":"ContainerStarted","Data":"a73b87856d00e4e556d884d1763f919ac5b8fed7bc50001ffaeaf41142b34aac"} Oct 30 13:01:02.345740 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:02.345715 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59337106-4z5sd" event={"ID":"5d8a2bcf-c3d1-4370-a603-1728b9958a5a","Type":"ContainerStarted","Data":"4d91e415e58c9b9895acb3f807befed79de7dd045be240542f8987c0c3967f07"} Oct 30 13:01:03.353030 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:03.352994 4931 generic.go:356] "Generic (PLEG): container finished" podID="5d8a2bcf-c3d1-4370-a603-1728b9958a5a" containerID="5d913bfbf01a84da37eda446c8a3acc2e61e3e95f4029a986a6dd04ff98cc52f" exitCode=0 Oct 30 13:01:03.353506 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:03.353084 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59337106-4z5sd" event={"ID":"5d8a2bcf-c3d1-4370-a603-1728b9958a5a","Type":"ContainerDied","Data":"5d913bfbf01a84da37eda446c8a3acc2e61e3e95f4029a986a6dd04ff98cc52f"} Oct 30 13:01:03.507362 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:03.507335 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-479d95c1-thznf" Oct 30 13:01:03.615986 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:03.615954 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/267c535f-c034-4727-8c4f-55c6565e5ffb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/267c535f-c034-4727-8c4f-55c6565e5ffb-maas-api-xks-ca\") pod \"267c535f-c034-4727-8c4f-55c6565e5ffb\" (UID: \"267c535f-c034-4727-8c4f-55c6565e5ffb\") " Oct 30 13:01:03.616166 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:03.616027 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/267c535f-c034-4727-8c4f-55c6565e5ffb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/267c535f-c034-4727-8c4f-55c6565e5ffb-maas-api-openshift-ca\") pod \"267c535f-c034-4727-8c4f-55c6565e5ffb\" (UID: \"267c535f-c034-4727-8c4f-55c6565e5ffb\") " Oct 30 13:01:03.616166 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:03.616068 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/267c535f-c034-4727-8c4f-55c6565e5ffb-kube-api-access-g4dpw\" (UniqueName: \"kubernetes.io/projected/267c535f-c034-4727-8c4f-55c6565e5ffb-kube-api-access-g4dpw\") pod \"267c535f-c034-4727-8c4f-55c6565e5ffb\" (UID: \"267c535f-c034-4727-8c4f-55c6565e5ffb\") " Oct 30 13:01:03.616992 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:03.616934 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/267c535f-c034-4727-8c4f-55c6565e5ffb-maas-api-openshift-ca" pod "267c535f-c034-4727-8c4f-55c6565e5ffb" (UID: "267c535f-c034-4727-8c4f-55c6565e5ffb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:01:03.618347 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:03.617665 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/267c535f-c034-4727-8c4f-55c6565e5ffb-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:01:03.625198 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:03.625173 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/267c535f-c034-4727-8c4f-55c6565e5ffb-kube-api-access-g4dpw" pod "267c535f-c034-4727-8c4f-55c6565e5ffb" (UID: "267c535f-c034-4727-8c4f-55c6565e5ffb"). InnerVolumeSpecName "kube-api-access-g4dpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:01:03.625372 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:03.625346 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/267c535f-c034-4727-8c4f-55c6565e5ffb-maas-api-xks-ca" pod "267c535f-c034-4727-8c4f-55c6565e5ffb" (UID: "267c535f-c034-4727-8c4f-55c6565e5ffb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:01:03.719140 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:03.719107 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4dpw\" (UniqueName: \"kubernetes.io/projected/267c535f-c034-4727-8c4f-55c6565e5ffb-kube-api-access-g4dpw\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:01:03.719140 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:03.719139 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/267c535f-c034-4727-8c4f-55c6565e5ffb-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:01:04.360041 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:04.360007 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-479d95c1-thznf" Oct 30 13:01:04.360482 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:04.360044 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-479d95c1-thznf" event={"ID":"267c535f-c034-4727-8c4f-55c6565e5ffb","Type":"ContainerDied","Data":"a73b87856d00e4e556d884d1763f919ac5b8fed7bc50001ffaeaf41142b34aac"} Oct 30 13:01:04.360482 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:04.360131 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a73b87856d00e4e556d884d1763f919ac5b8fed7bc50001ffaeaf41142b34aac" Oct 30 13:01:04.501518 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:04.501493 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59337106-4z5sd" Oct 30 13:01:04.630207 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:04.630133 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d8a2bcf-c3d1-4370-a603-1728b9958a5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d8a2bcf-c3d1-4370-a603-1728b9958a5a-maas-api-xks-ca\") pod \"5d8a2bcf-c3d1-4370-a603-1728b9958a5a\" (UID: \"5d8a2bcf-c3d1-4370-a603-1728b9958a5a\") " Oct 30 13:01:04.630207 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:04.630194 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d8a2bcf-c3d1-4370-a603-1728b9958a5a-kube-api-access-7s5gs\" (UniqueName: \"kubernetes.io/projected/5d8a2bcf-c3d1-4370-a603-1728b9958a5a-kube-api-access-7s5gs\") pod \"5d8a2bcf-c3d1-4370-a603-1728b9958a5a\" (UID: \"5d8a2bcf-c3d1-4370-a603-1728b9958a5a\") " Oct 30 13:01:04.630435 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:04.630311 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d8a2bcf-c3d1-4370-a603-1728b9958a5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d8a2bcf-c3d1-4370-a603-1728b9958a5a-maas-api-openshift-ca\") pod \"5d8a2bcf-c3d1-4370-a603-1728b9958a5a\" (UID: \"5d8a2bcf-c3d1-4370-a603-1728b9958a5a\") " Oct 30 13:01:04.630700 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:04.630674 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d8a2bcf-c3d1-4370-a603-1728b9958a5a-maas-api-openshift-ca" pod "5d8a2bcf-c3d1-4370-a603-1728b9958a5a" (UID: "5d8a2bcf-c3d1-4370-a603-1728b9958a5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:01:04.632306 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:04.632259 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d8a2bcf-c3d1-4370-a603-1728b9958a5a-maas-api-xks-ca" pod "5d8a2bcf-c3d1-4370-a603-1728b9958a5a" (UID: "5d8a2bcf-c3d1-4370-a603-1728b9958a5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:01:04.632306 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:04.632287 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d8a2bcf-c3d1-4370-a603-1728b9958a5a-kube-api-access-7s5gs" pod "5d8a2bcf-c3d1-4370-a603-1728b9958a5a" (UID: "5d8a2bcf-c3d1-4370-a603-1728b9958a5a"). InnerVolumeSpecName "kube-api-access-7s5gs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:01:04.731158 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:04.731125 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d8a2bcf-c3d1-4370-a603-1728b9958a5a-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:01:04.731158 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:04.731153 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s5gs\" (UniqueName: \"kubernetes.io/projected/5d8a2bcf-c3d1-4370-a603-1728b9958a5a-kube-api-access-7s5gs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:01:04.731158 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:04.731163 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d8a2bcf-c3d1-4370-a603-1728b9958a5a-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:01:05.367753 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:05.367728 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59337106-4z5sd" Oct 30 13:01:05.368149 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:05.367763 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59337106-4z5sd" event={"ID":"5d8a2bcf-c3d1-4370-a603-1728b9958a5a","Type":"ContainerDied","Data":"4d91e415e58c9b9895acb3f807befed79de7dd045be240542f8987c0c3967f07"} Oct 30 13:01:05.368149 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:05.367815 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d91e415e58c9b9895acb3f807befed79de7dd045be240542f8987c0c3967f07" Oct 30 13:01:07.351659 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:01:07.351620 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:01:07.680607 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:07.680522 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:01:09.522568 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.522535 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs"] Oct 30 13:01:09.523031 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.523016 4931 cpu_manager.go:414] "containerMap: removing container" podUID="267c535f-c034-4727-8c4f-55c6565e5ffb" containerName="revoke-keys" Oct 30 13:01:09.523031 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.523032 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="267c535f-c034-4727-8c4f-55c6565e5ffb" containerName="revoke-keys" Oct 30 13:01:09.523152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.523041 4931 cpu_manager.go:414] "containerMap: removing container" podUID="5d8a2bcf-c3d1-4370-a603-1728b9958a5a" containerName="revoke-keys" Oct 30 13:01:09.523152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.523046 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d8a2bcf-c3d1-4370-a603-1728b9958a5a" containerName="revoke-keys" Oct 30 13:01:09.523152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.523124 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="267c535f-c034-4727-8c4f-55c6565e5ffb" containerName="revoke-keys" Oct 30 13:01:09.523152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.523133 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d8a2bcf-c3d1-4370-a603-1728b9958a5a" containerName="revoke-keys" Oct 30 13:01:09.527390 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.527370 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" Oct 30 13:01:09.545054 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.545032 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-7a3603-openshift-default-dockercfg-dh98z\"" Oct 30 13:01:09.561567 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.561542 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs"] Oct 30 13:01:09.579756 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.579715 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/655adb9b-cbfb-476b-9c20-600e99055bd0-workload-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" Oct 30 13:01:09.579899 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.579810 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/655adb9b-cbfb-476b-9c20-600e99055bd0-istio-token\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" Oct 30 13:01:09.579899 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.579871 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/655adb9b-cbfb-476b-9c20-600e99055bd0-istio-data\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" Oct 30 13:01:09.580018 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.579908 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59p27\" (UniqueName: \"kubernetes.io/projected/655adb9b-cbfb-476b-9c20-600e99055bd0-kube-api-access-59p27\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" Oct 30 13:01:09.580018 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.579962 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/655adb9b-cbfb-476b-9c20-600e99055bd0-credential-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" Oct 30 13:01:09.580018 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.580002 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/655adb9b-cbfb-476b-9c20-600e99055bd0-istio-podinfo\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" Oct 30 13:01:09.580187 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.580031 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/655adb9b-cbfb-476b-9c20-600e99055bd0-istio-envoy\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" Oct 30 13:01:09.580187 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.580063 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/655adb9b-cbfb-476b-9c20-600e99055bd0-workload-certs\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" Oct 30 13:01:09.580187 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.580079 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/655adb9b-cbfb-476b-9c20-600e99055bd0-istiod-ca-cert\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" Oct 30 13:01:09.681463 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.681432 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59p27\" (UniqueName: \"kubernetes.io/projected/655adb9b-cbfb-476b-9c20-600e99055bd0-kube-api-access-59p27\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" Oct 30 13:01:09.681647 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.681483 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/655adb9b-cbfb-476b-9c20-600e99055bd0-credential-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" Oct 30 13:01:09.681647 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.681526 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/655adb9b-cbfb-476b-9c20-600e99055bd0-istio-podinfo\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" Oct 30 13:01:09.681647 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.681546 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/655adb9b-cbfb-476b-9c20-600e99055bd0-istio-envoy\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" Oct 30 13:01:09.681647 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.681568 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/655adb9b-cbfb-476b-9c20-600e99055bd0-workload-certs\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" Oct 30 13:01:09.681861 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.681728 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/655adb9b-cbfb-476b-9c20-600e99055bd0-istiod-ca-cert\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" Oct 30 13:01:09.681861 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.681804 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/655adb9b-cbfb-476b-9c20-600e99055bd0-workload-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" Oct 30 13:01:09.681974 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.681899 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/655adb9b-cbfb-476b-9c20-600e99055bd0-istio-token\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" Oct 30 13:01:09.682029 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.681963 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/655adb9b-cbfb-476b-9c20-600e99055bd0-workload-certs\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" Oct 30 13:01:09.682029 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.681976 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/655adb9b-cbfb-476b-9c20-600e99055bd0-istio-data\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" Oct 30 13:01:09.682145 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.682043 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/655adb9b-cbfb-476b-9c20-600e99055bd0-credential-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" Oct 30 13:01:09.682390 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.682365 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/655adb9b-cbfb-476b-9c20-600e99055bd0-workload-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" Oct 30 13:01:09.682501 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.682464 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/655adb9b-cbfb-476b-9c20-600e99055bd0-istio-data\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" Oct 30 13:01:09.682547 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.682515 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/655adb9b-cbfb-476b-9c20-600e99055bd0-istiod-ca-cert\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" Oct 30 13:01:09.684014 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.683984 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/655adb9b-cbfb-476b-9c20-600e99055bd0-istio-envoy\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" Oct 30 13:01:09.684169 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.684147 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/655adb9b-cbfb-476b-9c20-600e99055bd0-istio-podinfo\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" Oct 30 13:01:09.707338 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.707304 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59p27\" (UniqueName: \"kubernetes.io/projected/655adb9b-cbfb-476b-9c20-600e99055bd0-kube-api-access-59p27\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" Oct 30 13:01:09.707772 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.707750 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/655adb9b-cbfb-476b-9c20-600e99055bd0-istio-token\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" Oct 30 13:01:09.840992 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:09.840900 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" Oct 30 13:01:10.173193 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:10.173157 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs"] Oct 30 13:01:10.173973 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:01:10.173939 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod655adb9b_cbfb_476b_9c20_600e99055bd0.slice/crio-0deaddf130994a56d065855e6dbba80c9b9c8ee254ec4895ec3c012b98b0f5c9 WatchSource:0}: Error finding container 0deaddf130994a56d065855e6dbba80c9b9c8ee254ec4895ec3c012b98b0f5c9: Status 404 returned error can't find the container with id 0deaddf130994a56d065855e6dbba80c9b9c8ee254ec4895ec3c012b98b0f5c9 Oct 30 13:01:10.180893 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:10.180855 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:01:10.180992 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:10.180940 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:01:10.180992 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:10.180984 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:01:10.404332 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:10.404294 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" event={"ID":"655adb9b-cbfb-476b-9c20-600e99055bd0","Type":"ContainerStarted","Data":"226d7284288e4128bfe7b8d3647878631b7685596a9b06332a0566c4bf22783b"} Oct 30 13:01:10.404524 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:10.404351 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" event={"ID":"655adb9b-cbfb-476b-9c20-600e99055bd0","Type":"ContainerStarted","Data":"0deaddf130994a56d065855e6dbba80c9b9c8ee254ec4895ec3c012b98b0f5c9"} Oct 30 13:01:10.453361 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:10.453305 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" podStartSLOduration=1.453291808 podStartE2EDuration="1.453291808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:10.450994118 +0000 UTC m=+969.054492577" watchObservedRunningTime="2026-10-30 13:01:10.453291808 +0000 UTC m=+969.056790261" Oct 30 13:01:10.842111 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:10.842002 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" Oct 30 13:01:11.160398 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:11.160236 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7983812e-nqpd8"] Oct 30 13:01:11.191493 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:11.164631 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7983812e-nqpd8" Oct 30 13:01:11.191493 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:11.189027 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7983812e-nqpd8"] Oct 30 13:01:11.212293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:11.209938 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:01:11.212293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:11.209997 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:01:11.212293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:11.212013 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9cbs\" (UniqueName: \"kubernetes.io/projected/f69c855f-3d94-41af-baec-6fb3190c4f75-kube-api-access-p9cbs\") pod \"maas-api-revoke-sub-7983812e-nqpd8\" (UID: \"f69c855f-3d94-41af-baec-6fb3190c4f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7983812e-nqpd8" Oct 30 13:01:11.212293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:11.212287 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f69c855f-3d94-41af-baec-6fb3190c4f75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7983812e-nqpd8\" (UID: \"f69c855f-3d94-41af-baec-6fb3190c4f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7983812e-nqpd8" Oct 30 13:01:11.212662 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:11.212457 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f69c855f-3d94-41af-baec-6fb3190c4f75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7983812e-nqpd8\" (UID: \"f69c855f-3d94-41af-baec-6fb3190c4f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7983812e-nqpd8" Oct 30 13:01:11.314143 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:11.314101 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f69c855f-3d94-41af-baec-6fb3190c4f75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7983812e-nqpd8\" (UID: \"f69c855f-3d94-41af-baec-6fb3190c4f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7983812e-nqpd8" Oct 30 13:01:11.314376 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:11.314226 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f69c855f-3d94-41af-baec-6fb3190c4f75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7983812e-nqpd8\" (UID: \"f69c855f-3d94-41af-baec-6fb3190c4f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7983812e-nqpd8" Oct 30 13:01:11.314376 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:11.314319 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9cbs\" (UniqueName: \"kubernetes.io/projected/f69c855f-3d94-41af-baec-6fb3190c4f75-kube-api-access-p9cbs\") pod \"maas-api-revoke-sub-7983812e-nqpd8\" (UID: \"f69c855f-3d94-41af-baec-6fb3190c4f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7983812e-nqpd8" Oct 30 13:01:11.314863 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:11.314838 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f69c855f-3d94-41af-baec-6fb3190c4f75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7983812e-nqpd8\" (UID: \"f69c855f-3d94-41af-baec-6fb3190c4f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7983812e-nqpd8" Oct 30 13:01:11.316639 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:11.316620 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f69c855f-3d94-41af-baec-6fb3190c4f75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7983812e-nqpd8\" (UID: \"f69c855f-3d94-41af-baec-6fb3190c4f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7983812e-nqpd8" Oct 30 13:01:11.360551 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:11.360507 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9cbs\" (UniqueName: \"kubernetes.io/projected/f69c855f-3d94-41af-baec-6fb3190c4f75-kube-api-access-p9cbs\") pod \"maas-api-revoke-sub-7983812e-nqpd8\" (UID: \"f69c855f-3d94-41af-baec-6fb3190c4f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7983812e-nqpd8" Oct 30 13:01:11.482685 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:11.482647 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7983812e-nqpd8" Oct 30 13:01:11.859610 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:11.859494 4931 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 13:01:11.859610 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:11.859579 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" podUID="655adb9b-cbfb-476b-9c20-600e99055bd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 13:01:11.872653 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:01:11.872618 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf69c855f_3d94_41af_baec_6fb3190c4f75.slice/crio-d77fe818664420d35a7c9e5209ccf6fe85f8acd68e79775ab43aa339538b125c WatchSource:0}: Error finding container d77fe818664420d35a7c9e5209ccf6fe85f8acd68e79775ab43aa339538b125c: Status 404 returned error can't find the container with id d77fe818664420d35a7c9e5209ccf6fe85f8acd68e79775ab43aa339538b125c Oct 30 13:01:11.873714 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:11.873684 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7983812e-nqpd8"] Oct 30 13:01:12.417841 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:12.417803 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7983812e-nqpd8" event={"ID":"f69c855f-3d94-41af-baec-6fb3190c4f75","Type":"ContainerStarted","Data":"b00a6d043a35b67fb1520621cc56992f027891345508cb630d2d8a2b8ff16368"} Oct 30 13:01:12.418027 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:12.417864 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7983812e-nqpd8" event={"ID":"f69c855f-3d94-41af-baec-6fb3190c4f75","Type":"ContainerStarted","Data":"d77fe818664420d35a7c9e5209ccf6fe85f8acd68e79775ab43aa339538b125c"} Oct 30 13:01:12.462639 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:12.462587 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7983812e-nqpd8" podStartSLOduration=1.462572688 podStartE2EDuration="1.462572688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:12.461490674 +0000 UTC m=+971.064989120" watchObservedRunningTime="2026-10-30 13:01:12.462572688 +0000 UTC m=+971.066071134" Oct 30 13:01:12.847668 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:12.842208 4931 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 13:01:12.847668 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:12.842320 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" podUID="655adb9b-cbfb-476b-9c20-600e99055bd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 13:01:13.425196 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:13.425163 4931 generic.go:356] "Generic (PLEG): container finished" podID="f69c855f-3d94-41af-baec-6fb3190c4f75" containerID="b00a6d043a35b67fb1520621cc56992f027891345508cb630d2d8a2b8ff16368" exitCode=0 Oct 30 13:01:13.425636 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:13.425235 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7983812e-nqpd8" event={"ID":"f69c855f-3d94-41af-baec-6fb3190c4f75","Type":"ContainerDied","Data":"b00a6d043a35b67fb1520621cc56992f027891345508cb630d2d8a2b8ff16368"} Oct 30 13:01:13.852292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:13.848662 4931 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 13:01:13.852292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:13.848722 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" podUID="655adb9b-cbfb-476b-9c20-600e99055bd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 13:01:14.629456 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:14.629424 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7983812e-nqpd8" Oct 30 13:01:14.781044 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:14.781010 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f69c855f-3d94-41af-baec-6fb3190c4f75-kube-api-access-p9cbs\" (UniqueName: \"kubernetes.io/projected/f69c855f-3d94-41af-baec-6fb3190c4f75-kube-api-access-p9cbs\") pod \"f69c855f-3d94-41af-baec-6fb3190c4f75\" (UID: \"f69c855f-3d94-41af-baec-6fb3190c4f75\") " Oct 30 13:01:14.781253 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:14.781063 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f69c855f-3d94-41af-baec-6fb3190c4f75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f69c855f-3d94-41af-baec-6fb3190c4f75-maas-api-xks-ca\") pod \"f69c855f-3d94-41af-baec-6fb3190c4f75\" (UID: \"f69c855f-3d94-41af-baec-6fb3190c4f75\") " Oct 30 13:01:14.781253 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:14.781189 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f69c855f-3d94-41af-baec-6fb3190c4f75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f69c855f-3d94-41af-baec-6fb3190c4f75-maas-api-openshift-ca\") pod \"f69c855f-3d94-41af-baec-6fb3190c4f75\" (UID: \"f69c855f-3d94-41af-baec-6fb3190c4f75\") " Oct 30 13:01:14.781628 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:14.781597 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f69c855f-3d94-41af-baec-6fb3190c4f75-maas-api-openshift-ca" pod "f69c855f-3d94-41af-baec-6fb3190c4f75" (UID: "f69c855f-3d94-41af-baec-6fb3190c4f75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:01:14.783350 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:14.783325 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f69c855f-3d94-41af-baec-6fb3190c4f75-kube-api-access-p9cbs" pod "f69c855f-3d94-41af-baec-6fb3190c4f75" (UID: "f69c855f-3d94-41af-baec-6fb3190c4f75"). InnerVolumeSpecName "kube-api-access-p9cbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:01:14.784440 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:14.784415 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f69c855f-3d94-41af-baec-6fb3190c4f75-maas-api-xks-ca" pod "f69c855f-3d94-41af-baec-6fb3190c4f75" (UID: "f69c855f-3d94-41af-baec-6fb3190c4f75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:01:14.850831 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:14.850795 4931 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 13:01:14.851049 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:14.851025 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" podUID="655adb9b-cbfb-476b-9c20-600e99055bd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 13:01:14.883865 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:14.883832 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f69c855f-3d94-41af-baec-6fb3190c4f75-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:01:14.883865 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:14.883861 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9cbs\" (UniqueName: \"kubernetes.io/projected/f69c855f-3d94-41af-baec-6fb3190c4f75-kube-api-access-p9cbs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:01:14.883865 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:14.883871 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f69c855f-3d94-41af-baec-6fb3190c4f75-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:01:15.225131 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:15.225092 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8"] Oct 30 13:01:15.225580 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:15.225564 4931 cpu_manager.go:414] "containerMap: removing container" podUID="f69c855f-3d94-41af-baec-6fb3190c4f75" containerName="revoke-keys" Oct 30 13:01:15.225580 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:15.225578 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f69c855f-3d94-41af-baec-6fb3190c4f75" containerName="revoke-keys" Oct 30 13:01:15.225736 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:15.225645 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f69c855f-3d94-41af-baec-6fb3190c4f75" containerName="revoke-keys" Oct 30 13:01:15.228849 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:15.228832 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:01:15.263583 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:15.263466 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8"] Oct 30 13:01:15.263940 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:15.263921 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-67f56e\"/\"distinct-w4-67f56e-kserve-self-signed-certs\"" Oct 30 13:01:15.389626 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:15.389587 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsd6z\" (UniqueName: \"kubernetes.io/projected/5a504237-e5a3-491f-8f84-0419cf502f9a-kube-api-access-tsd6z\") pod \"distinct-w4-67f56e-kserve-6487fd96db-cl7z8\" (UID: \"5a504237-e5a3-491f-8f84-0419cf502f9a\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:01:15.389833 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:15.389649 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a504237-e5a3-491f-8f84-0419cf502f9a-home\") pod \"distinct-w4-67f56e-kserve-6487fd96db-cl7z8\" (UID: \"5a504237-e5a3-491f-8f84-0419cf502f9a\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:01:15.389833 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:15.389741 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a504237-e5a3-491f-8f84-0419cf502f9a-tls-certs\") pod \"distinct-w4-67f56e-kserve-6487fd96db-cl7z8\" (UID: \"5a504237-e5a3-491f-8f84-0419cf502f9a\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:01:15.389833 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:15.389766 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a504237-e5a3-491f-8f84-0419cf502f9a-model-cache\") pod \"distinct-w4-67f56e-kserve-6487fd96db-cl7z8\" (UID: \"5a504237-e5a3-491f-8f84-0419cf502f9a\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:01:15.390014 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:15.389864 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a504237-e5a3-491f-8f84-0419cf502f9a-tmp-dir\") pod \"distinct-w4-67f56e-kserve-6487fd96db-cl7z8\" (UID: \"5a504237-e5a3-491f-8f84-0419cf502f9a\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:01:15.390014 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:15.389918 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a504237-e5a3-491f-8f84-0419cf502f9a-dshm\") pod \"distinct-w4-67f56e-kserve-6487fd96db-cl7z8\" (UID: \"5a504237-e5a3-491f-8f84-0419cf502f9a\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:01:15.472290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:15.467337 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7983812e-nqpd8" event={"ID":"f69c855f-3d94-41af-baec-6fb3190c4f75","Type":"ContainerDied","Data":"d77fe818664420d35a7c9e5209ccf6fe85f8acd68e79775ab43aa339538b125c"} Oct 30 13:01:15.472290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:15.467440 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d77fe818664420d35a7c9e5209ccf6fe85f8acd68e79775ab43aa339538b125c" Oct 30 13:01:15.472290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:15.467511 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7983812e-nqpd8" Oct 30 13:01:15.490981 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:15.490887 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a504237-e5a3-491f-8f84-0419cf502f9a-home\") pod \"distinct-w4-67f56e-kserve-6487fd96db-cl7z8\" (UID: \"5a504237-e5a3-491f-8f84-0419cf502f9a\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:01:15.491237 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:15.491215 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a504237-e5a3-491f-8f84-0419cf502f9a-home\") pod \"distinct-w4-67f56e-kserve-6487fd96db-cl7z8\" (UID: \"5a504237-e5a3-491f-8f84-0419cf502f9a\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:01:15.491369 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:15.491257 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a504237-e5a3-491f-8f84-0419cf502f9a-tls-certs\") pod \"distinct-w4-67f56e-kserve-6487fd96db-cl7z8\" (UID: \"5a504237-e5a3-491f-8f84-0419cf502f9a\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:01:15.491369 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:15.491293 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a504237-e5a3-491f-8f84-0419cf502f9a-model-cache\") pod \"distinct-w4-67f56e-kserve-6487fd96db-cl7z8\" (UID: \"5a504237-e5a3-491f-8f84-0419cf502f9a\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:01:15.491369 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:15.491334 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a504237-e5a3-491f-8f84-0419cf502f9a-tmp-dir\") pod \"distinct-w4-67f56e-kserve-6487fd96db-cl7z8\" (UID: \"5a504237-e5a3-491f-8f84-0419cf502f9a\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:01:15.491369 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:15.491352 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a504237-e5a3-491f-8f84-0419cf502f9a-dshm\") pod \"distinct-w4-67f56e-kserve-6487fd96db-cl7z8\" (UID: \"5a504237-e5a3-491f-8f84-0419cf502f9a\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:01:15.491586 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:15.491438 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsd6z\" (UniqueName: \"kubernetes.io/projected/5a504237-e5a3-491f-8f84-0419cf502f9a-kube-api-access-tsd6z\") pod \"distinct-w4-67f56e-kserve-6487fd96db-cl7z8\" (UID: \"5a504237-e5a3-491f-8f84-0419cf502f9a\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:01:15.491974 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:15.491752 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a504237-e5a3-491f-8f84-0419cf502f9a-tmp-dir\") pod \"distinct-w4-67f56e-kserve-6487fd96db-cl7z8\" (UID: \"5a504237-e5a3-491f-8f84-0419cf502f9a\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:01:15.491974 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:15.491937 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a504237-e5a3-491f-8f84-0419cf502f9a-model-cache\") pod \"distinct-w4-67f56e-kserve-6487fd96db-cl7z8\" (UID: \"5a504237-e5a3-491f-8f84-0419cf502f9a\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:01:15.494678 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:15.494652 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a504237-e5a3-491f-8f84-0419cf502f9a-tls-certs\") pod \"distinct-w4-67f56e-kserve-6487fd96db-cl7z8\" (UID: \"5a504237-e5a3-491f-8f84-0419cf502f9a\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:01:15.494877 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:15.494852 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a504237-e5a3-491f-8f84-0419cf502f9a-dshm\") pod \"distinct-w4-67f56e-kserve-6487fd96db-cl7z8\" (UID: \"5a504237-e5a3-491f-8f84-0419cf502f9a\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:01:15.518283 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:15.517697 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsd6z\" (UniqueName: \"kubernetes.io/projected/5a504237-e5a3-491f-8f84-0419cf502f9a-kube-api-access-tsd6z\") pod \"distinct-w4-67f56e-kserve-6487fd96db-cl7z8\" (UID: \"5a504237-e5a3-491f-8f84-0419cf502f9a\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:01:15.566650 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:15.566612 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:01:15.844788 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:15.844689 4931 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 13:01:15.845378 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:15.845345 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" podUID="655adb9b-cbfb-476b-9c20-600e99055bd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 13:01:15.924624 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:15.924597 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8"] Oct 30 13:01:15.926351 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:01:15.926322 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a504237_e5a3_491f_8f84_0419cf502f9a.slice/crio-1930ef76186fc36438946a99dcf84ebe19e8f3cfef0782b33617bba1819a2be3 WatchSource:0}: Error finding container 1930ef76186fc36438946a99dcf84ebe19e8f3cfef0782b33617bba1819a2be3: Status 404 returned error can't find the container with id 1930ef76186fc36438946a99dcf84ebe19e8f3cfef0782b33617bba1819a2be3 Oct 30 13:01:16.474804 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:16.474767 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" event={"ID":"5a504237-e5a3-491f-8f84-0419cf502f9a","Type":"ContainerStarted","Data":"87de98e52fbad25f10a8eed84ca4beb9ffad207d4a2d5745c82dd84cc5d416a6"} Oct 30 13:01:16.474804 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:16.474810 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" event={"ID":"5a504237-e5a3-491f-8f84-0419cf502f9a","Type":"ContainerStarted","Data":"1930ef76186fc36438946a99dcf84ebe19e8f3cfef0782b33617bba1819a2be3"} Oct 30 13:01:16.522547 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:16.522497 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" podStartSLOduration=1.522483527 podStartE2EDuration="1.522483527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:16.521871041 +0000 UTC m=+975.125369488" watchObservedRunningTime="2026-10-30 13:01:16.522483527 +0000 UTC m=+975.125981972" Oct 30 13:01:16.567203 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:16.567167 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:01:16.841659 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:16.841577 4931 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 13:01:16.841659 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:16.841632 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" podUID="655adb9b-cbfb-476b-9c20-600e99055bd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 13:01:17.845796 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:17.845767 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" Oct 30 13:01:17.846228 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:17.846178 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" Oct 30 13:01:17.846945 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:17.846923 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" Oct 30 13:01:24.765140 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:24.765102 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6"] Oct 30 13:01:24.769029 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:24.769008 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" Oct 30 13:01:24.787611 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:24.787591 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-c2ce75-openshift-default-dockercfg-44wkt\"" Oct 30 13:01:24.798334 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:24.798308 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29a14a0f-98aa-4981-abe4-56919051950b-istio-envoy\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" Oct 30 13:01:24.798461 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:24.798340 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29a14a0f-98aa-4981-abe4-56919051950b-istiod-ca-cert\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" Oct 30 13:01:24.798461 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:24.798378 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a14a0f-98aa-4981-abe4-56919051950b-credential-socket\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" Oct 30 13:01:24.798581 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:24.798465 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29a14a0f-98aa-4981-abe4-56919051950b-istio-podinfo\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" Oct 30 13:01:24.798581 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:24.798522 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29a14a0f-98aa-4981-abe4-56919051950b-istio-token\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" Oct 30 13:01:24.798581 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:24.798542 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqfnl\" (UniqueName: \"kubernetes.io/projected/29a14a0f-98aa-4981-abe4-56919051950b-kube-api-access-rqfnl\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" Oct 30 13:01:24.798581 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:24.798569 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29a14a0f-98aa-4981-abe4-56919051950b-workload-certs\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" Oct 30 13:01:24.798783 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:24.798718 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29a14a0f-98aa-4981-abe4-56919051950b-istio-data\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" Oct 30 13:01:24.798841 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:24.798783 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a14a0f-98aa-4981-abe4-56919051950b-workload-socket\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" Oct 30 13:01:24.818425 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:24.818340 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6"] Oct 30 13:01:24.899362 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:24.899325 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29a14a0f-98aa-4981-abe4-56919051950b-istio-token\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" Oct 30 13:01:24.899572 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:24.899373 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqfnl\" (UniqueName: \"kubernetes.io/projected/29a14a0f-98aa-4981-abe4-56919051950b-kube-api-access-rqfnl\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" Oct 30 13:01:24.899572 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:24.899413 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29a14a0f-98aa-4981-abe4-56919051950b-workload-certs\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" Oct 30 13:01:24.899572 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:24.899482 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29a14a0f-98aa-4981-abe4-56919051950b-istio-data\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" Oct 30 13:01:24.899572 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:24.899530 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a14a0f-98aa-4981-abe4-56919051950b-workload-socket\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" Oct 30 13:01:24.899572 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:24.899568 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29a14a0f-98aa-4981-abe4-56919051950b-istio-envoy\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" Oct 30 13:01:24.899869 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:24.899591 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29a14a0f-98aa-4981-abe4-56919051950b-istiod-ca-cert\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" Oct 30 13:01:24.899869 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:24.899638 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a14a0f-98aa-4981-abe4-56919051950b-credential-socket\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" Oct 30 13:01:24.899869 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:24.899705 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29a14a0f-98aa-4981-abe4-56919051950b-istio-podinfo\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" Oct 30 13:01:24.900000 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:24.899943 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29a14a0f-98aa-4981-abe4-56919051950b-workload-certs\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" Oct 30 13:01:24.900000 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:24.899967 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a14a0f-98aa-4981-abe4-56919051950b-workload-socket\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" Oct 30 13:01:24.900158 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:24.900136 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29a14a0f-98aa-4981-abe4-56919051950b-istio-data\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" Oct 30 13:01:24.900348 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:24.900321 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a14a0f-98aa-4981-abe4-56919051950b-credential-socket\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" Oct 30 13:01:24.900847 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:24.900820 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29a14a0f-98aa-4981-abe4-56919051950b-istiod-ca-cert\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" Oct 30 13:01:24.902075 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:24.902042 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29a14a0f-98aa-4981-abe4-56919051950b-istio-envoy\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" Oct 30 13:01:24.902320 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:24.902302 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29a14a0f-98aa-4981-abe4-56919051950b-istio-podinfo\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" Oct 30 13:01:24.924043 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:24.924018 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29a14a0f-98aa-4981-abe4-56919051950b-istio-token\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" Oct 30 13:01:24.924174 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:24.924152 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqfnl\" (UniqueName: \"kubernetes.io/projected/29a14a0f-98aa-4981-abe4-56919051950b-kube-api-access-rqfnl\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" Oct 30 13:01:25.082005 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.081895 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" Oct 30 13:01:25.343389 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.343260 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md"] Oct 30 13:01:25.347519 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.347491 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" Oct 30 13:01:25.402299 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.399001 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md"] Oct 30 13:01:25.433453 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.433409 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6"] Oct 30 13:01:25.433968 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:01:25.433939 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a14a0f_98aa_4981_abe4_56919051950b.slice/crio-a5b5192d4486af132a3562fe9624516d6742b1bae928f9594744baeb05340f6c WatchSource:0}: Error finding container a5b5192d4486af132a3562fe9624516d6742b1bae928f9594744baeb05340f6c: Status 404 returned error can't find the container with id a5b5192d4486af132a3562fe9624516d6742b1bae928f9594744baeb05340f6c Oct 30 13:01:25.441509 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.438722 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:01:25.441509 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.438802 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:01:25.441509 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.438846 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:01:25.507834 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.507799 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-credential-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" Oct 30 13:01:25.507967 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.507912 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-istio-envoy\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" Oct 30 13:01:25.507967 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.507942 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-workload-certs\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" Oct 30 13:01:25.508095 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.507996 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvcx4\" (UniqueName: \"kubernetes.io/projected/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-kube-api-access-mvcx4\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" Oct 30 13:01:25.508095 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.508016 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-istio-token\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" Oct 30 13:01:25.508095 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.508068 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-istio-podinfo\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" Oct 30 13:01:25.508095 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.508089 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-istio-data\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" Oct 30 13:01:25.508378 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.508109 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-workload-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" Oct 30 13:01:25.508378 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.508166 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-istiod-ca-cert\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" Oct 30 13:01:25.533328 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.533289 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" event={"ID":"29a14a0f-98aa-4981-abe4-56919051950b","Type":"ContainerStarted","Data":"18c18901545c45e7aff743ac7204a95e8d926847e5e6e82f30222c81deadd3a4"} Oct 30 13:01:25.533518 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.533357 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" event={"ID":"29a14a0f-98aa-4981-abe4-56919051950b","Type":"ContainerStarted","Data":"a5b5192d4486af132a3562fe9624516d6742b1bae928f9594744baeb05340f6c"} Oct 30 13:01:25.567478 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.567441 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:01:25.580703 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.580656 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:01:25.586122 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.586060 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" podStartSLOduration=1.586041196 podStartE2EDuration="1.586041196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:25.580732982 +0000 UTC m=+984.184231429" watchObservedRunningTime="2026-10-30 13:01:25.586041196 +0000 UTC m=+984.189539642" Oct 30 13:01:25.609129 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.609044 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvcx4\" (UniqueName: \"kubernetes.io/projected/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-kube-api-access-mvcx4\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" Oct 30 13:01:25.609129 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.609092 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-istio-token\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" Oct 30 13:01:25.609367 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.609158 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-istio-podinfo\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" Oct 30 13:01:25.609367 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.609189 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-istio-data\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" Oct 30 13:01:25.609367 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.609218 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-workload-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" Oct 30 13:01:25.609367 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.609339 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-istiod-ca-cert\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" Oct 30 13:01:25.609757 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.609740 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-credential-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" Oct 30 13:01:25.610152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.609668 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-istio-data\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" Oct 30 13:01:25.610152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.609809 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-istio-envoy\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" Oct 30 13:01:25.610152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.609839 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-workload-certs\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" Oct 30 13:01:25.610152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.609948 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-workload-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" Oct 30 13:01:25.610152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.610038 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-credential-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" Oct 30 13:01:25.610351 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.610157 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-istiod-ca-cert\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" Oct 30 13:01:25.610351 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.610202 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-workload-certs\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" Oct 30 13:01:25.611849 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.611829 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-istio-podinfo\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" Oct 30 13:01:25.612451 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.612435 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-istio-envoy\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" Oct 30 13:01:25.633964 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.633925 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-istio-token\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" Oct 30 13:01:25.634368 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.634348 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvcx4\" (UniqueName: \"kubernetes.io/projected/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-kube-api-access-mvcx4\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" Oct 30 13:01:25.712667 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:25.712474 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" Oct 30 13:01:26.076408 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:26.076371 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md"] Oct 30 13:01:26.084783 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:26.084675 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" Oct 30 13:01:26.085529 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:01:26.085488 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e8d41cb_daa2_4be3_b42d_90a1951a1fc3.slice/crio-7528eeae935e63884fd3e887a8b1e95b305b21c0cc07a3aff8a8cf7408971af7 WatchSource:0}: Error finding container 7528eeae935e63884fd3e887a8b1e95b305b21c0cc07a3aff8a8cf7408971af7: Status 404 returned error can't find the container with id 7528eeae935e63884fd3e887a8b1e95b305b21c0cc07a3aff8a8cf7408971af7 Oct 30 13:01:26.086841 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:26.086450 4931 patch_prober.go:29] interesting pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 13:01:26.086841 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:26.086487 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" podUID="29a14a0f-98aa-4981-abe4-56919051950b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 13:01:26.088918 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:26.088889 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:01:26.089065 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:26.089049 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:01:26.089161 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:26.089149 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:01:26.563642 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:26.563605 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" event={"ID":"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3","Type":"ContainerStarted","Data":"960d1dc95143fa3ac61a3addd1f220bf7b0de4ab1b412036f6e99cb1f293d796"} Oct 30 13:01:26.564212 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:26.563662 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" event={"ID":"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3","Type":"ContainerStarted","Data":"7528eeae935e63884fd3e887a8b1e95b305b21c0cc07a3aff8a8cf7408971af7"} Oct 30 13:01:26.580418 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:26.580390 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:01:26.613507 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:26.613454 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" podStartSLOduration=1.6134377340000001 podStartE2EDuration="1.613437734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:26.611807728 +0000 UTC m=+985.215306174" watchObservedRunningTime="2026-10-30 13:01:26.613437734 +0000 UTC m=+985.216936238" Oct 30 13:01:26.713027 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:26.712994 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" Oct 30 13:01:27.082764 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:27.082724 4931 patch_prober.go:29] interesting pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 13:01:27.084305 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:27.083355 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" podUID="29a14a0f-98aa-4981-abe4-56919051950b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 13:01:27.714712 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:27.714659 4931 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 13:01:27.714915 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:27.714727 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" podUID="2e8d41cb-daa2-4be3-b42d-90a1951a1fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 13:01:28.082883 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:28.082759 4931 patch_prober.go:29] interesting pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 13:01:28.082883 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:28.082854 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" podUID="29a14a0f-98aa-4981-abe4-56919051950b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 13:01:28.713194 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:28.713160 4931 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 13:01:28.713391 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:28.713209 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" podUID="2e8d41cb-daa2-4be3-b42d-90a1951a1fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 13:01:29.082830 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:29.082730 4931 patch_prober.go:29] interesting pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 13:01:29.082830 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:29.082783 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" podUID="29a14a0f-98aa-4981-abe4-56919051950b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 13:01:29.713540 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:29.713508 4931 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 13:01:29.713793 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:29.713766 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" podUID="2e8d41cb-daa2-4be3-b42d-90a1951a1fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 13:01:30.082982 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:30.082686 4931 patch_prober.go:29] interesting pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 13:01:30.082982 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:30.082743 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" podUID="29a14a0f-98aa-4981-abe4-56919051950b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 13:01:30.713670 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:30.713632 4931 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 13:01:30.713831 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:30.713691 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" podUID="2e8d41cb-daa2-4be3-b42d-90a1951a1fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 13:01:31.082885 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:31.082780 4931 patch_prober.go:29] interesting pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 13:01:31.083177 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:31.083130 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" podUID="29a14a0f-98aa-4981-abe4-56919051950b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 13:01:31.713459 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:31.713410 4931 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 13:01:31.713656 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:31.713470 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" podUID="2e8d41cb-daa2-4be3-b42d-90a1951a1fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 13:01:32.082914 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:32.082833 4931 patch_prober.go:29] interesting pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 13:01:32.082914 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:32.082892 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" podUID="29a14a0f-98aa-4981-abe4-56919051950b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 13:01:32.713469 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:32.713428 4931 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 13:01:32.713946 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:32.713496 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" podUID="2e8d41cb-daa2-4be3-b42d-90a1951a1fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 13:01:33.087801 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:33.087708 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" Oct 30 13:01:33.088166 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:33.088145 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" Oct 30 13:01:33.088872 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:33.088854 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" Oct 30 13:01:33.718535 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:33.718503 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" Oct 30 13:01:33.719039 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:33.718878 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" Oct 30 13:01:33.719556 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:33.719537 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" Oct 30 13:01:37.353476 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:01:37.353437 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:01:40.673471 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:40.673436 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:01:54.247397 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.247362 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn"] Oct 30 13:01:54.251523 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.251496 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" Oct 30 13:01:54.267930 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.267909 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-09db16-openshift-default-dockercfg-l5hf4\"" Oct 30 13:01:54.285604 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.285576 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn"] Oct 30 13:01:54.397543 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.397508 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42c653e7-3421-4d55-b035-33d6f286413a-istio-data\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" Oct 30 13:01:54.397543 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.397549 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42c653e7-3421-4d55-b035-33d6f286413a-istio-token\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" Oct 30 13:01:54.397770 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.397567 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42c653e7-3421-4d55-b035-33d6f286413a-istio-podinfo\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" Oct 30 13:01:54.397770 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.397639 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c653e7-3421-4d55-b035-33d6f286413a-workload-socket\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" Oct 30 13:01:54.397770 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.397706 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42c653e7-3421-4d55-b035-33d6f286413a-istio-envoy\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" Oct 30 13:01:54.397770 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.397743 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42c653e7-3421-4d55-b035-33d6f286413a-istiod-ca-cert\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" Oct 30 13:01:54.397770 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.397764 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42c653e7-3421-4d55-b035-33d6f286413a-workload-certs\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" Oct 30 13:01:54.398030 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.397947 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c653e7-3421-4d55-b035-33d6f286413a-credential-socket\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" Oct 30 13:01:54.398030 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.398011 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcphr\" (UniqueName: \"kubernetes.io/projected/42c653e7-3421-4d55-b035-33d6f286413a-kube-api-access-tcphr\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" Oct 30 13:01:54.498798 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.498721 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42c653e7-3421-4d55-b035-33d6f286413a-istiod-ca-cert\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" Oct 30 13:01:54.498798 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.498768 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42c653e7-3421-4d55-b035-33d6f286413a-workload-certs\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" Oct 30 13:01:54.499015 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.498831 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c653e7-3421-4d55-b035-33d6f286413a-credential-socket\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" Oct 30 13:01:54.499015 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.498871 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcphr\" (UniqueName: \"kubernetes.io/projected/42c653e7-3421-4d55-b035-33d6f286413a-kube-api-access-tcphr\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" Oct 30 13:01:54.499015 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.498917 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42c653e7-3421-4d55-b035-33d6f286413a-istio-data\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" Oct 30 13:01:54.499015 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.498942 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42c653e7-3421-4d55-b035-33d6f286413a-istio-token\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" Oct 30 13:01:54.499015 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.498963 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42c653e7-3421-4d55-b035-33d6f286413a-istio-podinfo\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" Oct 30 13:01:54.499015 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.498988 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c653e7-3421-4d55-b035-33d6f286413a-workload-socket\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" Oct 30 13:01:54.499349 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.499048 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42c653e7-3421-4d55-b035-33d6f286413a-istio-envoy\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" Oct 30 13:01:54.499349 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.499257 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c653e7-3421-4d55-b035-33d6f286413a-credential-socket\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" Oct 30 13:01:54.499437 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.499351 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42c653e7-3421-4d55-b035-33d6f286413a-istio-data\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" Oct 30 13:01:54.499525 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.499507 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c653e7-3421-4d55-b035-33d6f286413a-workload-socket\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" Oct 30 13:01:54.499588 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.499568 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42c653e7-3421-4d55-b035-33d6f286413a-istiod-ca-cert\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" Oct 30 13:01:54.499687 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.499672 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42c653e7-3421-4d55-b035-33d6f286413a-workload-certs\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" Oct 30 13:01:54.501412 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.501388 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42c653e7-3421-4d55-b035-33d6f286413a-istio-envoy\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" Oct 30 13:01:54.501585 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.501564 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42c653e7-3421-4d55-b035-33d6f286413a-istio-podinfo\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" Oct 30 13:01:54.523716 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.523689 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42c653e7-3421-4d55-b035-33d6f286413a-istio-token\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" Oct 30 13:01:54.523929 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.523907 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcphr\" (UniqueName: \"kubernetes.io/projected/42c653e7-3421-4d55-b035-33d6f286413a-kube-api-access-tcphr\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" Oct 30 13:01:54.564732 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.564710 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" Oct 30 13:01:54.888922 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.888888 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn"] Oct 30 13:01:54.889087 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:01:54.888933 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42c653e7_3421_4d55_b035_33d6f286413a.slice/crio-650e4f21a82e7439efdc7c1ae32bbf65f4b1a698522b8c1e78228cda28503de8 WatchSource:0}: Error finding container 650e4f21a82e7439efdc7c1ae32bbf65f4b1a698522b8c1e78228cda28503de8: Status 404 returned error can't find the container with id 650e4f21a82e7439efdc7c1ae32bbf65f4b1a698522b8c1e78228cda28503de8 Oct 30 13:01:54.891307 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.891261 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:01:54.891384 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.891342 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:01:54.891384 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:54.891371 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:01:55.041934 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.041891 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8"] Oct 30 13:01:55.046540 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.046518 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" Oct 30 13:01:55.062924 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.062903 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-67f56e-openshift-default-dockercfg-vvpqd\"" Oct 30 13:01:55.098733 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.098697 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8"] Oct 30 13:01:55.105094 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.105062 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e1ed676-cd86-4b17-97af-1a5a501f6274-istio-token\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" Oct 30 13:01:55.105249 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.105120 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e1ed676-cd86-4b17-97af-1a5a501f6274-workload-certs\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" Oct 30 13:01:55.105249 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.105175 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wznc8\" (UniqueName: \"kubernetes.io/projected/4e1ed676-cd86-4b17-97af-1a5a501f6274-kube-api-access-wznc8\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" Oct 30 13:01:55.105249 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.105224 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e1ed676-cd86-4b17-97af-1a5a501f6274-istio-podinfo\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" Oct 30 13:01:55.105474 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.105260 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1ed676-cd86-4b17-97af-1a5a501f6274-workload-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" Oct 30 13:01:55.105474 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.105305 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e1ed676-cd86-4b17-97af-1a5a501f6274-istio-data\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" Oct 30 13:01:55.105474 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.105359 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e1ed676-cd86-4b17-97af-1a5a501f6274-istiod-ca-cert\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" Oct 30 13:01:55.105474 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.105404 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e1ed676-cd86-4b17-97af-1a5a501f6274-istio-envoy\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" Oct 30 13:01:55.105474 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.105431 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1ed676-cd86-4b17-97af-1a5a501f6274-credential-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" Oct 30 13:01:55.186772 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.186677 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8"] Oct 30 13:01:55.190605 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.190583 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" Oct 30 13:01:55.205974 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.205937 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e1ed676-cd86-4b17-97af-1a5a501f6274-istio-envoy\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" Oct 30 13:01:55.206096 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.205980 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1ed676-cd86-4b17-97af-1a5a501f6274-credential-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" Oct 30 13:01:55.206096 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.206022 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc65620b-073b-4e13-82da-38526fb90da3-istiod-ca-cert\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" Oct 30 13:01:55.206182 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.206092 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e1ed676-cd86-4b17-97af-1a5a501f6274-istio-token\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" Oct 30 13:01:55.206182 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.206118 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc65620b-073b-4e13-82da-38526fb90da3-istio-envoy\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" Oct 30 13:01:55.206182 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.206162 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e1ed676-cd86-4b17-97af-1a5a501f6274-workload-certs\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" Oct 30 13:01:55.206378 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.206256 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc65620b-073b-4e13-82da-38526fb90da3-credential-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" Oct 30 13:01:55.206378 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.206303 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc65620b-073b-4e13-82da-38526fb90da3-workload-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" Oct 30 13:01:55.206378 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.206331 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc65620b-073b-4e13-82da-38526fb90da3-workload-certs\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" Oct 30 13:01:55.206535 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.206377 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wznc8\" (UniqueName: \"kubernetes.io/projected/4e1ed676-cd86-4b17-97af-1a5a501f6274-kube-api-access-wznc8\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" Oct 30 13:01:55.206669 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.206647 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1ed676-cd86-4b17-97af-1a5a501f6274-credential-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" Oct 30 13:01:55.206736 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.206692 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lm54\" (UniqueName: \"kubernetes.io/projected/fc65620b-073b-4e13-82da-38526fb90da3-kube-api-access-8lm54\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" Oct 30 13:01:55.206736 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.206722 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc65620b-073b-4e13-82da-38526fb90da3-istio-data\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" Oct 30 13:01:55.206836 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.206753 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e1ed676-cd86-4b17-97af-1a5a501f6274-istio-podinfo\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" Oct 30 13:01:55.206836 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.206785 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc65620b-073b-4e13-82da-38526fb90da3-istio-token\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" Oct 30 13:01:55.206836 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.206792 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e1ed676-cd86-4b17-97af-1a5a501f6274-workload-certs\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" Oct 30 13:01:55.206836 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.206811 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1ed676-cd86-4b17-97af-1a5a501f6274-workload-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" Oct 30 13:01:55.207053 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.206846 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e1ed676-cd86-4b17-97af-1a5a501f6274-istio-data\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" Oct 30 13:01:55.207053 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.206862 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc65620b-073b-4e13-82da-38526fb90da3-istio-podinfo\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" Oct 30 13:01:55.207053 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.206902 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e1ed676-cd86-4b17-97af-1a5a501f6274-istiod-ca-cert\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" Oct 30 13:01:55.207345 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.207322 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e1ed676-cd86-4b17-97af-1a5a501f6274-istio-data\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" Oct 30 13:01:55.207625 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.207604 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1ed676-cd86-4b17-97af-1a5a501f6274-workload-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" Oct 30 13:01:55.207706 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.207625 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e1ed676-cd86-4b17-97af-1a5a501f6274-istiod-ca-cert\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" Oct 30 13:01:55.208636 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.208617 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e1ed676-cd86-4b17-97af-1a5a501f6274-istio-envoy\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" Oct 30 13:01:55.209358 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.209336 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e1ed676-cd86-4b17-97af-1a5a501f6274-istio-podinfo\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" Oct 30 13:01:55.211814 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.211788 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-8b46c9-openshift-default-dockercfg-9flrx\"" Oct 30 13:01:55.216118 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.216099 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8"] Oct 30 13:01:55.232631 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.232581 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e1ed676-cd86-4b17-97af-1a5a501f6274-istio-token\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" Oct 30 13:01:55.233018 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.232998 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wznc8\" (UniqueName: \"kubernetes.io/projected/4e1ed676-cd86-4b17-97af-1a5a501f6274-kube-api-access-wznc8\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" Oct 30 13:01:55.307545 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.307512 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc65620b-073b-4e13-82da-38526fb90da3-istiod-ca-cert\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" Oct 30 13:01:55.308040 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.307577 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc65620b-073b-4e13-82da-38526fb90da3-istio-envoy\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" Oct 30 13:01:55.308040 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.307626 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc65620b-073b-4e13-82da-38526fb90da3-credential-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" Oct 30 13:01:55.308040 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.307652 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc65620b-073b-4e13-82da-38526fb90da3-workload-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" Oct 30 13:01:55.308040 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.307785 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc65620b-073b-4e13-82da-38526fb90da3-workload-certs\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" Oct 30 13:01:55.308040 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.307850 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lm54\" (UniqueName: \"kubernetes.io/projected/fc65620b-073b-4e13-82da-38526fb90da3-kube-api-access-8lm54\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" Oct 30 13:01:55.308040 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.307888 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc65620b-073b-4e13-82da-38526fb90da3-istio-data\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" Oct 30 13:01:55.308040 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.307934 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc65620b-073b-4e13-82da-38526fb90da3-istio-token\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" Oct 30 13:01:55.308040 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.307978 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc65620b-073b-4e13-82da-38526fb90da3-istio-podinfo\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" Oct 30 13:01:55.308040 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.308031 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc65620b-073b-4e13-82da-38526fb90da3-credential-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" Oct 30 13:01:55.308520 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.308059 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc65620b-073b-4e13-82da-38526fb90da3-workload-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" Oct 30 13:01:55.308520 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.308373 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc65620b-073b-4e13-82da-38526fb90da3-istiod-ca-cert\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" Oct 30 13:01:55.308520 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.308433 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc65620b-073b-4e13-82da-38526fb90da3-istio-data\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" Oct 30 13:01:55.308774 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.308749 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc65620b-073b-4e13-82da-38526fb90da3-workload-certs\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" Oct 30 13:01:55.310567 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.310545 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc65620b-073b-4e13-82da-38526fb90da3-istio-envoy\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" Oct 30 13:01:55.310800 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.310782 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc65620b-073b-4e13-82da-38526fb90da3-istio-podinfo\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" Oct 30 13:01:55.332824 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.332799 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lm54\" (UniqueName: \"kubernetes.io/projected/fc65620b-073b-4e13-82da-38526fb90da3-kube-api-access-8lm54\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" Oct 30 13:01:55.333126 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.333106 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc65620b-073b-4e13-82da-38526fb90da3-istio-token\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" Oct 30 13:01:55.362999 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.362975 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" Oct 30 13:01:55.506098 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.506064 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" Oct 30 13:01:55.691413 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.691383 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8"] Oct 30 13:01:55.693136 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:01:55.693073 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e1ed676_cd86_4b17_97af_1a5a501f6274.slice/crio-f67d6ab3a49497ea1b6aad79a42570e0a5b34fc91c0ed758fb7f3eeaed8452db WatchSource:0}: Error finding container f67d6ab3a49497ea1b6aad79a42570e0a5b34fc91c0ed758fb7f3eeaed8452db: Status 404 returned error can't find the container with id f67d6ab3a49497ea1b6aad79a42570e0a5b34fc91c0ed758fb7f3eeaed8452db Oct 30 13:01:55.695669 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.695633 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:01:55.695782 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.695715 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:01:55.695782 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.695757 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:01:55.802886 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.802851 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" event={"ID":"4e1ed676-cd86-4b17-97af-1a5a501f6274","Type":"ContainerStarted","Data":"f67d6ab3a49497ea1b6aad79a42570e0a5b34fc91c0ed758fb7f3eeaed8452db"} Oct 30 13:01:55.804627 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.804601 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" event={"ID":"42c653e7-3421-4d55-b035-33d6f286413a","Type":"ContainerStarted","Data":"5b2cc422b1fedef338f30defec154e0f7289e8cc859731d96f7acfa5c82520b8"} Oct 30 13:01:55.804744 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.804645 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" event={"ID":"42c653e7-3421-4d55-b035-33d6f286413a","Type":"ContainerStarted","Data":"650e4f21a82e7439efdc7c1ae32bbf65f4b1a698522b8c1e78228cda28503de8"} Oct 30 13:01:55.814628 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.813952 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8"] Oct 30 13:01:55.815573 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:01:55.815548 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc65620b_073b_4e13_82da_38526fb90da3.slice/crio-e972492a39d424bd03f26bee52131d022e19afd40ac6d4722f666ee464de5f85 WatchSource:0}: Error finding container e972492a39d424bd03f26bee52131d022e19afd40ac6d4722f666ee464de5f85: Status 404 returned error can't find the container with id e972492a39d424bd03f26bee52131d022e19afd40ac6d4722f666ee464de5f85 Oct 30 13:01:55.817768 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.817736 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:01:55.817864 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.817811 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:01:55.817864 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:55.817856 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:01:56.573292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:56.572363 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" Oct 30 13:01:56.580833 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:56.580798 4931 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 13:01:56.581090 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:56.581062 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" podUID="42c653e7-3421-4d55-b035-33d6f286413a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 13:01:56.818469 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:56.818411 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" event={"ID":"fc65620b-073b-4e13-82da-38526fb90da3","Type":"ContainerStarted","Data":"decf2dabff7f6ac00617d6984a38c747dafc1a911633f93c4f2b5cef67483bae"} Oct 30 13:01:56.818653 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:56.818493 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" event={"ID":"fc65620b-073b-4e13-82da-38526fb90da3","Type":"ContainerStarted","Data":"e972492a39d424bd03f26bee52131d022e19afd40ac6d4722f666ee464de5f85"} Oct 30 13:01:56.820393 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:56.820368 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" event={"ID":"4e1ed676-cd86-4b17-97af-1a5a501f6274","Type":"ContainerStarted","Data":"38fa2303b37360ec91dc36073c63945b29064d92b92fab921da816d78b386456"} Oct 30 13:01:56.879351 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:56.874310 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" podStartSLOduration=2.8742897640000002 podStartE2EDuration="2.874289764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:55.865008428 +0000 UTC m=+1014.468506879" watchObservedRunningTime="2026-10-30 13:01:56.874289764 +0000 UTC m=+1015.477788213" Oct 30 13:01:56.903294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:56.901098 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" podStartSLOduration=1.901081199 podStartE2EDuration="1.901081199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:56.880701831 +0000 UTC m=+1015.484200294" watchObservedRunningTime="2026-10-30 13:01:56.901081199 +0000 UTC m=+1015.504579647" Oct 30 13:01:56.959290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:56.958900 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" podStartSLOduration=1.958878042 podStartE2EDuration="1.958878042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:56.955649384 +0000 UTC m=+1015.559147832" watchObservedRunningTime="2026-10-30 13:01:56.958878042 +0000 UTC m=+1015.562376489" Oct 30 13:01:57.365875 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:57.365836 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" Oct 30 13:01:57.373428 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:57.373388 4931 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 13:01:57.373710 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:57.373676 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" podUID="4e1ed676-cd86-4b17-97af-1a5a501f6274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 13:01:57.508737 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:57.508430 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" Oct 30 13:01:57.510724 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:57.510685 4931 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 13:01:57.510976 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:57.510932 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" podUID="fc65620b-073b-4e13-82da-38526fb90da3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 13:01:57.567920 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:57.567873 4931 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 13:01:57.568104 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:57.567941 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" podUID="42c653e7-3421-4d55-b035-33d6f286413a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 13:01:58.367293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:58.366616 4931 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 13:01:58.367293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:58.366680 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" podUID="4e1ed676-cd86-4b17-97af-1a5a501f6274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 13:01:58.511291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:58.507397 4931 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 13:01:58.511291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:58.507472 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" podUID="fc65620b-073b-4e13-82da-38526fb90da3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 13:01:58.566291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:58.565351 4931 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 13:01:58.566291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:58.565415 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" podUID="42c653e7-3421-4d55-b035-33d6f286413a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 13:01:59.364744 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:59.364689 4931 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 13:01:59.365164 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:59.365128 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" podUID="4e1ed676-cd86-4b17-97af-1a5a501f6274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 13:01:59.510229 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:59.509667 4931 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 13:01:59.510229 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:59.509726 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" podUID="fc65620b-073b-4e13-82da-38526fb90da3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 13:01:59.582291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:59.581563 4931 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 13:01:59.582291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:01:59.581625 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" podUID="42c653e7-3421-4d55-b035-33d6f286413a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 13:02:00.367292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:00.367165 4931 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 13:02:00.367292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:00.367226 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" podUID="4e1ed676-cd86-4b17-97af-1a5a501f6274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 13:02:00.508667 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:00.508628 4931 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 13:02:00.508947 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:00.508915 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" podUID="fc65620b-073b-4e13-82da-38526fb90da3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 13:02:00.576326 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:00.576289 4931 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 13:02:00.576936 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:00.576896 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" podUID="42c653e7-3421-4d55-b035-33d6f286413a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 13:02:01.373291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:01.364649 4931 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 13:02:01.373291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:01.364716 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" podUID="4e1ed676-cd86-4b17-97af-1a5a501f6274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 13:02:01.507878 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:01.507841 4931 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 13:02:01.508236 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:01.508209 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" podUID="fc65620b-073b-4e13-82da-38526fb90da3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 13:02:01.568290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:01.567670 4931 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 13:02:01.568290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:01.567761 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" podUID="42c653e7-3421-4d55-b035-33d6f286413a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 13:02:02.363728 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:02.363686 4931 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 13:02:02.364250 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:02.363756 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" podUID="4e1ed676-cd86-4b17-97af-1a5a501f6274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 13:02:02.506518 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:02.506478 4931 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 13:02:02.506733 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:02.506543 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" podUID="fc65620b-073b-4e13-82da-38526fb90da3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 13:02:02.570573 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:02.570538 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" Oct 30 13:02:02.571062 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:02.571032 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" Oct 30 13:02:02.571904 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:02.571886 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" Oct 30 13:02:03.514879 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:03.514852 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" Oct 30 13:02:03.515394 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:03.515379 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" Oct 30 13:02:03.516280 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:03.516227 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" Oct 30 13:02:04.405441 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:04.405396 4931 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:02:04.405657 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:04.405486 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" podUID="4e1ed676-cd86-4b17-97af-1a5a501f6274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:02:04.538967 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:04.538931 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" Oct 30 13:02:04.539424 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:04.539331 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" Oct 30 13:02:04.540702 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:04.540675 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" Oct 30 13:02:05.788019 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:05.787981 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:02:07.354127 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:02:07.354085 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:02:08.583293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:08.583248 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80c0ff30-jmnwl"] Oct 30 13:02:08.588719 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:08.588698 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c0ff30-jmnwl" Oct 30 13:02:08.609013 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:08.608983 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80c0ff30-jmnwl"] Oct 30 13:02:08.617347 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:08.617330 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:02:08.618189 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:08.618165 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:02:08.711068 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:08.711038 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j72cw\" (UniqueName: \"kubernetes.io/projected/1b32e46d-cec9-4453-aebc-f4f45301146c-kube-api-access-j72cw\") pod \"maas-api-revoke-sub-80c0ff30-jmnwl\" (UID: \"1b32e46d-cec9-4453-aebc-f4f45301146c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c0ff30-jmnwl" Oct 30 13:02:08.711215 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:08.711105 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b32e46d-cec9-4453-aebc-f4f45301146c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80c0ff30-jmnwl\" (UID: \"1b32e46d-cec9-4453-aebc-f4f45301146c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c0ff30-jmnwl" Oct 30 13:02:08.711261 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:08.711212 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b32e46d-cec9-4453-aebc-f4f45301146c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80c0ff30-jmnwl\" (UID: \"1b32e46d-cec9-4453-aebc-f4f45301146c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c0ff30-jmnwl" Oct 30 13:02:08.812760 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:08.812726 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j72cw\" (UniqueName: \"kubernetes.io/projected/1b32e46d-cec9-4453-aebc-f4f45301146c-kube-api-access-j72cw\") pod \"maas-api-revoke-sub-80c0ff30-jmnwl\" (UID: \"1b32e46d-cec9-4453-aebc-f4f45301146c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c0ff30-jmnwl" Oct 30 13:02:08.812929 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:08.812783 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b32e46d-cec9-4453-aebc-f4f45301146c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80c0ff30-jmnwl\" (UID: \"1b32e46d-cec9-4453-aebc-f4f45301146c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c0ff30-jmnwl" Oct 30 13:02:08.812987 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:08.812966 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b32e46d-cec9-4453-aebc-f4f45301146c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80c0ff30-jmnwl\" (UID: \"1b32e46d-cec9-4453-aebc-f4f45301146c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c0ff30-jmnwl" Oct 30 13:02:08.813445 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:08.813423 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b32e46d-cec9-4453-aebc-f4f45301146c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80c0ff30-jmnwl\" (UID: \"1b32e46d-cec9-4453-aebc-f4f45301146c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c0ff30-jmnwl" Oct 30 13:02:08.815216 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:08.815200 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b32e46d-cec9-4453-aebc-f4f45301146c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80c0ff30-jmnwl\" (UID: \"1b32e46d-cec9-4453-aebc-f4f45301146c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c0ff30-jmnwl" Oct 30 13:02:08.837732 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:08.837676 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j72cw\" (UniqueName: \"kubernetes.io/projected/1b32e46d-cec9-4453-aebc-f4f45301146c-kube-api-access-j72cw\") pod \"maas-api-revoke-sub-80c0ff30-jmnwl\" (UID: \"1b32e46d-cec9-4453-aebc-f4f45301146c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c0ff30-jmnwl" Oct 30 13:02:08.902457 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:08.902430 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c0ff30-jmnwl" Oct 30 13:02:09.206666 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:09.206641 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80c0ff30-jmnwl"] Oct 30 13:02:09.207547 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:02:09.207527 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b32e46d_cec9_4453_aebc_f4f45301146c.slice/crio-827c7041f2052981c694c8c20d2ae759d0314a58e089da1218c460e369cf444f WatchSource:0}: Error finding container 827c7041f2052981c694c8c20d2ae759d0314a58e089da1218c460e369cf444f: Status 404 returned error can't find the container with id 827c7041f2052981c694c8c20d2ae759d0314a58e089da1218c460e369cf444f Oct 30 13:02:09.497537 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:09.497465 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2"] Oct 30 13:02:09.502815 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:09.502795 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" Oct 30 13:02:09.522658 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:09.522635 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-dd90cb-openshift-default-dockercfg-9nwsm\"" Oct 30 13:02:09.549876 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:09.549847 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2"] Oct 30 13:02:09.622282 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:09.622227 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-istio-token\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" Oct 30 13:02:09.622762 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:09.622309 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-workload-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" Oct 30 13:02:09.622762 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:09.622421 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-credential-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" Oct 30 13:02:09.622762 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:09.622449 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-workload-certs\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" Oct 30 13:02:09.622762 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:09.622498 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-istio-data\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" Oct 30 13:02:09.622762 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:09.622528 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-istiod-ca-cert\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" Oct 30 13:02:09.622762 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:09.622593 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-istio-podinfo\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" Oct 30 13:02:09.623052 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:09.622821 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7h2s\" (UniqueName: \"kubernetes.io/projected/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-kube-api-access-s7h2s\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" Oct 30 13:02:09.623052 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:09.622881 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-istio-envoy\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" Oct 30 13:02:09.723677 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:09.723634 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7h2s\" (UniqueName: \"kubernetes.io/projected/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-kube-api-access-s7h2s\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" Oct 30 13:02:09.723677 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:09.723685 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-istio-envoy\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" Oct 30 13:02:09.723945 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:09.723741 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-istio-token\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" Oct 30 13:02:09.723945 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:09.723763 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-workload-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" Oct 30 13:02:09.723945 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:09.723797 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-credential-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" Oct 30 13:02:09.723945 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:09.723814 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-workload-certs\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" Oct 30 13:02:09.723945 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:09.723842 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-istio-data\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" Oct 30 13:02:09.723945 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:09.723866 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-istiod-ca-cert\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" Oct 30 13:02:09.723945 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:09.723923 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-istio-podinfo\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" Oct 30 13:02:09.724700 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:09.724220 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-workload-certs\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" Oct 30 13:02:09.724700 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:09.724287 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-credential-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" Oct 30 13:02:09.724700 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:09.724394 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-workload-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" Oct 30 13:02:09.724700 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:09.724612 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-istio-data\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" Oct 30 13:02:09.724866 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:09.724740 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-istiod-ca-cert\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" Oct 30 13:02:09.726161 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:09.726137 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-istio-envoy\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" Oct 30 13:02:09.726400 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:09.726382 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-istio-podinfo\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" Oct 30 13:02:09.761611 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:09.761540 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-istio-token\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" Oct 30 13:02:09.762290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:09.762255 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7h2s\" (UniqueName: \"kubernetes.io/projected/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-kube-api-access-s7h2s\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" Oct 30 13:02:09.818960 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:09.818938 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" Oct 30 13:02:09.965793 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:09.965753 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c0ff30-jmnwl" event={"ID":"1b32e46d-cec9-4453-aebc-f4f45301146c","Type":"ContainerStarted","Data":"0c5a08b31a7003fee8b00d03db9fc585b25c137ac93eab886b901cb389935a74"} Oct 30 13:02:09.965963 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:09.965846 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c0ff30-jmnwl" event={"ID":"1b32e46d-cec9-4453-aebc-f4f45301146c","Type":"ContainerStarted","Data":"827c7041f2052981c694c8c20d2ae759d0314a58e089da1218c460e369cf444f"} Oct 30 13:02:10.012415 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:10.012324 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c0ff30-jmnwl" podStartSLOduration=2.012311213 podStartE2EDuration="2.012311213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:10.011088903 +0000 UTC m=+1028.614587346" watchObservedRunningTime="2026-10-30 13:02:10.012311213 +0000 UTC m=+1028.615809659" Oct 30 13:02:10.169699 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:02:10.169668 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3a9c9f2_eaac_4fcc_88e7_87d311d77677.slice/crio-60d0d8b9354cd986d31027673571709fea97eec15479bde577b022015e9f8b6c WatchSource:0}: Error finding container 60d0d8b9354cd986d31027673571709fea97eec15479bde577b022015e9f8b6c: Status 404 returned error can't find the container with id 60d0d8b9354cd986d31027673571709fea97eec15479bde577b022015e9f8b6c Oct 30 13:02:10.170851 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:10.170822 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2"] Oct 30 13:02:10.172185 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:10.172153 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:10.172318 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:10.172226 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:10.172318 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:10.172301 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:10.974183 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:10.974146 4931 generic.go:356] "Generic (PLEG): container finished" podID="1b32e46d-cec9-4453-aebc-f4f45301146c" containerID="0c5a08b31a7003fee8b00d03db9fc585b25c137ac93eab886b901cb389935a74" exitCode=0 Oct 30 13:02:10.974723 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:10.974233 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c0ff30-jmnwl" event={"ID":"1b32e46d-cec9-4453-aebc-f4f45301146c","Type":"ContainerDied","Data":"0c5a08b31a7003fee8b00d03db9fc585b25c137ac93eab886b901cb389935a74"} Oct 30 13:02:10.976096 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:10.976072 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" event={"ID":"d3a9c9f2-eaac-4fcc-88e7-87d311d77677","Type":"ContainerStarted","Data":"1ea9cf76063beed45e1a77f55f59edba70264cb55533b7cf544982535626f98c"} Oct 30 13:02:10.976259 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:10.976111 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" event={"ID":"d3a9c9f2-eaac-4fcc-88e7-87d311d77677","Type":"ContainerStarted","Data":"60d0d8b9354cd986d31027673571709fea97eec15479bde577b022015e9f8b6c"} Oct 30 13:02:11.084413 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:11.084362 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" podStartSLOduration=2.084345732 podStartE2EDuration="2.084345732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:11.081474784 +0000 UTC m=+1029.684973229" watchObservedRunningTime="2026-10-30 13:02:11.084345732 +0000 UTC m=+1029.687844177" Oct 30 13:02:11.820114 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:11.820079 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" Oct 30 13:02:11.821664 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:11.821637 4931 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 13:02:11.821811 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:11.821690 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" podUID="d3a9c9f2-eaac-4fcc-88e7-87d311d77677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 13:02:12.166364 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:12.166338 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c0ff30-jmnwl" Oct 30 13:02:12.272028 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:12.271989 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b32e46d-cec9-4453-aebc-f4f45301146c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b32e46d-cec9-4453-aebc-f4f45301146c-maas-api-openshift-ca\") pod \"1b32e46d-cec9-4453-aebc-f4f45301146c\" (UID: \"1b32e46d-cec9-4453-aebc-f4f45301146c\") " Oct 30 13:02:12.272220 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:12.272160 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b32e46d-cec9-4453-aebc-f4f45301146c-kube-api-access-j72cw\" (UniqueName: \"kubernetes.io/projected/1b32e46d-cec9-4453-aebc-f4f45301146c-kube-api-access-j72cw\") pod \"1b32e46d-cec9-4453-aebc-f4f45301146c\" (UID: \"1b32e46d-cec9-4453-aebc-f4f45301146c\") " Oct 30 13:02:12.272312 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:12.272257 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b32e46d-cec9-4453-aebc-f4f45301146c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b32e46d-cec9-4453-aebc-f4f45301146c-maas-api-xks-ca\") pod \"1b32e46d-cec9-4453-aebc-f4f45301146c\" (UID: \"1b32e46d-cec9-4453-aebc-f4f45301146c\") " Oct 30 13:02:12.272606 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:12.272581 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b32e46d-cec9-4453-aebc-f4f45301146c-maas-api-openshift-ca" pod "1b32e46d-cec9-4453-aebc-f4f45301146c" (UID: "1b32e46d-cec9-4453-aebc-f4f45301146c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:02:12.272807 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:12.272785 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b32e46d-cec9-4453-aebc-f4f45301146c-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:02:12.275303 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:12.275190 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b32e46d-cec9-4453-aebc-f4f45301146c-maas-api-xks-ca" pod "1b32e46d-cec9-4453-aebc-f4f45301146c" (UID: "1b32e46d-cec9-4453-aebc-f4f45301146c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:02:12.275478 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:12.275459 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b32e46d-cec9-4453-aebc-f4f45301146c-kube-api-access-j72cw" pod "1b32e46d-cec9-4453-aebc-f4f45301146c" (UID: "1b32e46d-cec9-4453-aebc-f4f45301146c"). InnerVolumeSpecName "kube-api-access-j72cw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:12.374302 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:12.374182 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j72cw\" (UniqueName: \"kubernetes.io/projected/1b32e46d-cec9-4453-aebc-f4f45301146c-kube-api-access-j72cw\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:02:12.374302 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:12.374226 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b32e46d-cec9-4453-aebc-f4f45301146c-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:02:12.819386 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:12.819351 4931 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 13:02:12.819561 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:12.819403 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" podUID="d3a9c9f2-eaac-4fcc-88e7-87d311d77677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 13:02:12.989366 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:12.989290 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c0ff30-jmnwl" Oct 30 13:02:12.989366 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:12.989310 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c0ff30-jmnwl" event={"ID":"1b32e46d-cec9-4453-aebc-f4f45301146c","Type":"ContainerDied","Data":"827c7041f2052981c694c8c20d2ae759d0314a58e089da1218c460e369cf444f"} Oct 30 13:02:12.989366 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:12.989358 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="827c7041f2052981c694c8c20d2ae759d0314a58e089da1218c460e369cf444f" Oct 30 13:02:13.820169 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:13.820133 4931 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 13:02:13.820565 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:13.820186 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" podUID="d3a9c9f2-eaac-4fcc-88e7-87d311d77677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 13:02:14.819925 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:14.819886 4931 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 13:02:14.820101 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:14.819956 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" podUID="d3a9c9f2-eaac-4fcc-88e7-87d311d77677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 13:02:15.819577 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:15.819544 4931 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 13:02:15.819980 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:15.819592 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" podUID="d3a9c9f2-eaac-4fcc-88e7-87d311d77677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 13:02:17.866289 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:17.865431 4931 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:02:17.866289 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:17.865519 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" podUID="d3a9c9f2-eaac-4fcc-88e7-87d311d77677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:02:17.908012 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:17.907986 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" Oct 30 13:02:17.908573 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:17.908546 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" Oct 30 13:02:17.909453 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:17.909430 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" Oct 30 13:02:26.323920 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:26.323886 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:02:37.353925 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:02:37.353876 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:02:40.213161 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.213127 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh"] Oct 30 13:02:40.213697 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.213681 4931 cpu_manager.go:414] "containerMap: removing container" podUID="1b32e46d-cec9-4453-aebc-f4f45301146c" containerName="revoke-keys" Oct 30 13:02:40.213742 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.213698 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b32e46d-cec9-4453-aebc-f4f45301146c" containerName="revoke-keys" Oct 30 13:02:40.213817 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.213808 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b32e46d-cec9-4453-aebc-f4f45301146c" containerName="revoke-keys" Oct 30 13:02:40.218587 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.218569 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" Oct 30 13:02:40.246238 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.246202 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh"] Oct 30 13:02:40.372049 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.372013 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn"] Oct 30 13:02:40.376257 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.376234 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" Oct 30 13:02:40.383701 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.383670 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-istio-podinfo\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" Oct 30 13:02:40.383897 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.383722 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-istiod-ca-cert\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" Oct 30 13:02:40.384096 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.383985 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-istio-token\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" Oct 30 13:02:40.384167 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.384096 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-istio-envoy\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" Oct 30 13:02:40.384222 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.384173 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-workload-certs\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" Oct 30 13:02:40.384222 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.384211 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-istio-data\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" Oct 30 13:02:40.384341 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.384240 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-credential-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" Oct 30 13:02:40.384341 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.384302 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-workload-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" Oct 30 13:02:40.384341 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.384324 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8s68\" (UniqueName: \"kubernetes.io/projected/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-kube-api-access-c8s68\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" Oct 30 13:02:40.404603 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.404557 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn"] Oct 30 13:02:40.485054 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.484970 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3680a3d2-0b13-40df-aeeb-6a924495f8d2-workload-certs\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" Oct 30 13:02:40.485225 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.485082 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-workload-certs\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" Oct 30 13:02:40.485225 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.485120 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-istio-data\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" Oct 30 13:02:40.485225 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.485140 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3680a3d2-0b13-40df-aeeb-6a924495f8d2-istio-podinfo\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" Oct 30 13:02:40.485225 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.485169 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-credential-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" Oct 30 13:02:40.485225 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.485185 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3680a3d2-0b13-40df-aeeb-6a924495f8d2-istio-token\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" Oct 30 13:02:40.485225 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.485218 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-workload-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" Oct 30 13:02:40.485549 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.485241 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3680a3d2-0b13-40df-aeeb-6a924495f8d2-istio-data\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" Oct 30 13:02:40.485549 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.485405 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8s68\" (UniqueName: \"kubernetes.io/projected/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-kube-api-access-c8s68\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" Oct 30 13:02:40.485549 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.485488 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9srd\" (UniqueName: \"kubernetes.io/projected/3680a3d2-0b13-40df-aeeb-6a924495f8d2-kube-api-access-z9srd\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" Oct 30 13:02:40.485729 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.485544 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-workload-certs\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" Oct 30 13:02:40.485729 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.485574 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-istio-podinfo\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" Oct 30 13:02:40.485729 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.485579 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-credential-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" Oct 30 13:02:40.485729 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.485624 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-istiod-ca-cert\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" Oct 30 13:02:40.485729 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.485624 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-istio-data\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" Oct 30 13:02:40.485729 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.485689 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3680a3d2-0b13-40df-aeeb-6a924495f8d2-istiod-ca-cert\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" Oct 30 13:02:40.485729 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.485697 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-workload-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" Oct 30 13:02:40.485978 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.485781 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3680a3d2-0b13-40df-aeeb-6a924495f8d2-credential-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" Oct 30 13:02:40.485978 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.485807 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3680a3d2-0b13-40df-aeeb-6a924495f8d2-istio-envoy\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" Oct 30 13:02:40.485978 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.485847 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-istio-token\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" Oct 30 13:02:40.485978 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.485895 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-istio-envoy\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" Oct 30 13:02:40.485978 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.485937 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3680a3d2-0b13-40df-aeeb-6a924495f8d2-workload-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" Oct 30 13:02:40.486220 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.486200 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-istiod-ca-cert\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" Oct 30 13:02:40.487988 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.487968 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-istio-envoy\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" Oct 30 13:02:40.488093 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.488074 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-istio-podinfo\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" Oct 30 13:02:40.514810 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.514778 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8s68\" (UniqueName: \"kubernetes.io/projected/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-kube-api-access-c8s68\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" Oct 30 13:02:40.518617 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.518589 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-istio-token\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" Oct 30 13:02:40.539584 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.539557 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" Oct 30 13:02:40.587466 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.587430 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3680a3d2-0b13-40df-aeeb-6a924495f8d2-workload-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" Oct 30 13:02:40.587654 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.587474 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3680a3d2-0b13-40df-aeeb-6a924495f8d2-workload-certs\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" Oct 30 13:02:40.587654 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.587570 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3680a3d2-0b13-40df-aeeb-6a924495f8d2-istio-podinfo\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" Oct 30 13:02:40.587654 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.587611 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3680a3d2-0b13-40df-aeeb-6a924495f8d2-istio-token\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" Oct 30 13:02:40.588106 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.587660 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3680a3d2-0b13-40df-aeeb-6a924495f8d2-istio-data\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" Oct 30 13:02:40.588106 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.587725 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9srd\" (UniqueName: \"kubernetes.io/projected/3680a3d2-0b13-40df-aeeb-6a924495f8d2-kube-api-access-z9srd\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" Oct 30 13:02:40.588106 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.587817 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3680a3d2-0b13-40df-aeeb-6a924495f8d2-istiod-ca-cert\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" Oct 30 13:02:40.588106 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.587873 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3680a3d2-0b13-40df-aeeb-6a924495f8d2-credential-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" Oct 30 13:02:40.588106 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.587888 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3680a3d2-0b13-40df-aeeb-6a924495f8d2-workload-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" Oct 30 13:02:40.588106 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.587899 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3680a3d2-0b13-40df-aeeb-6a924495f8d2-istio-envoy\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" Oct 30 13:02:40.588106 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.588068 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3680a3d2-0b13-40df-aeeb-6a924495f8d2-workload-certs\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" Oct 30 13:02:40.588360 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.588136 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3680a3d2-0b13-40df-aeeb-6a924495f8d2-istio-data\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" Oct 30 13:02:40.588360 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.588215 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3680a3d2-0b13-40df-aeeb-6a924495f8d2-credential-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" Oct 30 13:02:40.588639 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.588616 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3680a3d2-0b13-40df-aeeb-6a924495f8d2-istiod-ca-cert\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" Oct 30 13:02:40.590066 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.590043 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3680a3d2-0b13-40df-aeeb-6a924495f8d2-istio-envoy\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" Oct 30 13:02:40.590794 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.590767 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3680a3d2-0b13-40df-aeeb-6a924495f8d2-istio-podinfo\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" Oct 30 13:02:40.617442 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.617380 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9srd\" (UniqueName: \"kubernetes.io/projected/3680a3d2-0b13-40df-aeeb-6a924495f8d2-kube-api-access-z9srd\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" Oct 30 13:02:40.618834 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.618803 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3680a3d2-0b13-40df-aeeb-6a924495f8d2-istio-token\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" Oct 30 13:02:40.691212 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.691184 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" Oct 30 13:02:40.861712 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.861688 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh"] Oct 30 13:02:40.862996 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:02:40.862973 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d1bbe1_f3d7_4670_9bc7_b27dfe1e9eed.slice/crio-4be40406fef651a76fc35884a2b0fd884110373168dd4ce7e2561f73b6a1868e WatchSource:0}: Error finding container 4be40406fef651a76fc35884a2b0fd884110373168dd4ce7e2561f73b6a1868e: Status 404 returned error can't find the container with id 4be40406fef651a76fc35884a2b0fd884110373168dd4ce7e2561f73b6a1868e Oct 30 13:02:40.865423 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.865394 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:40.865499 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.865482 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:40.865540 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:40.865513 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:41.006446 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:41.006416 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn"] Oct 30 13:02:41.008045 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:02:41.008016 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3680a3d2_0b13_40df_aeeb_6a924495f8d2.slice/crio-7643248a06a0e4ada7f90a17182efbf2ee07de735b207c89cb732bffbfbb96ab WatchSource:0}: Error finding container 7643248a06a0e4ada7f90a17182efbf2ee07de735b207c89cb732bffbfbb96ab: Status 404 returned error can't find the container with id 7643248a06a0e4ada7f90a17182efbf2ee07de735b207c89cb732bffbfbb96ab Oct 30 13:02:41.010667 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:41.010632 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:41.010742 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:41.010700 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:41.010742 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:41.010727 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:41.180346 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:41.179546 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" event={"ID":"3680a3d2-0b13-40df-aeeb-6a924495f8d2","Type":"ContainerStarted","Data":"9bce9341e990823fb376085bb78badbdae08554f3cfa22fa99036200228d18b0"} Oct 30 13:02:41.180346 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:41.179633 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" event={"ID":"3680a3d2-0b13-40df-aeeb-6a924495f8d2","Type":"ContainerStarted","Data":"7643248a06a0e4ada7f90a17182efbf2ee07de735b207c89cb732bffbfbb96ab"} Oct 30 13:02:41.182058 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:41.182017 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" event={"ID":"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed","Type":"ContainerStarted","Data":"08442ed9c67d02f34f5c98e5886ef7b41e83df17b029b435c8d57ddfbc2dd4b0"} Oct 30 13:02:41.182210 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:41.182076 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" event={"ID":"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed","Type":"ContainerStarted","Data":"4be40406fef651a76fc35884a2b0fd884110373168dd4ce7e2561f73b6a1868e"} Oct 30 13:02:41.243425 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:41.243359 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" podStartSLOduration=1.243339081 podStartE2EDuration="1.243339081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:41.238016067 +0000 UTC m=+1059.841514514" watchObservedRunningTime="2026-10-30 13:02:41.243339081 +0000 UTC m=+1059.846837528" Oct 30 13:02:41.292582 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:41.292433 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" podStartSLOduration=1.29241101 podStartE2EDuration="1.29241101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:41.290280288 +0000 UTC m=+1059.893778726" watchObservedRunningTime="2026-10-30 13:02:41.29241101 +0000 UTC m=+1059.895909458" Oct 30 13:02:41.547315 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:41.547195 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" Oct 30 13:02:41.547667 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:41.547637 4931 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 13:02:41.547846 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:41.547823 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" podUID="16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 13:02:41.691713 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:41.691674 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" Oct 30 13:02:42.543631 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:42.543594 4931 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 13:02:42.544239 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:42.544203 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" podUID="16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 13:02:42.693009 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:42.692972 4931 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 13:02:42.693186 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:42.693025 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" podUID="3680a3d2-0b13-40df-aeeb-6a924495f8d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 13:02:43.555291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:43.554615 4931 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 13:02:43.555291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:43.554673 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" podUID="16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 13:02:43.692005 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:43.691969 4931 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 13:02:43.692193 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:43.692020 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" podUID="3680a3d2-0b13-40df-aeeb-6a924495f8d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 13:02:44.555292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:44.540649 4931 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 13:02:44.555292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:44.540709 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" podUID="16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 13:02:44.691777 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:44.691736 4931 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 13:02:44.691957 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:44.691807 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" podUID="3680a3d2-0b13-40df-aeeb-6a924495f8d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 13:02:45.567291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:45.557528 4931 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 13:02:45.567291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:45.557619 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" podUID="16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 13:02:45.693195 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:45.693160 4931 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 13:02:45.693432 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:45.693217 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" podUID="3680a3d2-0b13-40df-aeeb-6a924495f8d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 13:02:46.546820 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:46.546783 4931 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 13:02:46.547152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:46.547109 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" podUID="16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 13:02:46.692192 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:46.692013 4931 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 13:02:46.692192 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:46.692083 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" podUID="3680a3d2-0b13-40df-aeeb-6a924495f8d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 13:02:47.540073 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:47.540041 4931 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 13:02:47.540285 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:47.540103 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" podUID="16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 13:02:47.691788 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:47.691751 4931 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 13:02:47.691944 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:47.691820 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" podUID="3680a3d2-0b13-40df-aeeb-6a924495f8d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 13:02:48.545517 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:48.545480 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" Oct 30 13:02:48.545944 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:48.545923 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" Oct 30 13:02:48.547061 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:48.547036 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" Oct 30 13:02:48.721451 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:48.721416 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" Oct 30 13:02:48.723565 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:48.723527 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" Oct 30 13:02:48.729203 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:48.729172 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" Oct 30 13:02:55.630606 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:55.630572 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj"] Oct 30 13:02:55.635078 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:55.635048 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" Oct 30 13:02:55.660757 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:55.660717 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj"] Oct 30 13:02:55.771923 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:55.771885 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-credential-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" Oct 30 13:02:55.772085 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:55.771952 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk5wq\" (UniqueName: \"kubernetes.io/projected/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-kube-api-access-fk5wq\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" Oct 30 13:02:55.772085 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:55.771990 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-istio-podinfo\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" Oct 30 13:02:55.772085 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:55.772015 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-istio-envoy\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" Oct 30 13:02:55.772214 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:55.772164 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-istio-token\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" Oct 30 13:02:55.772281 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:55.772254 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-istiod-ca-cert\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" Oct 30 13:02:55.772321 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:55.772298 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-istio-data\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" Oct 30 13:02:55.772437 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:55.772421 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-workload-certs\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" Oct 30 13:02:55.772481 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:55.772469 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-workload-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" Oct 30 13:02:55.873851 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:55.873811 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-istiod-ca-cert\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" Oct 30 13:02:55.873851 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:55.873855 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-istio-data\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" Oct 30 13:02:55.874115 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:55.873925 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-workload-certs\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" Oct 30 13:02:55.874115 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:55.873978 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-workload-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" Oct 30 13:02:55.874115 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:55.874036 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-credential-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" Oct 30 13:02:55.874115 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:55.874088 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk5wq\" (UniqueName: \"kubernetes.io/projected/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-kube-api-access-fk5wq\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" Oct 30 13:02:55.874319 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:55.874143 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-istio-podinfo\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" Oct 30 13:02:55.874319 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:55.874171 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-istio-envoy\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" Oct 30 13:02:55.874319 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:55.874224 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-istio-token\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" Oct 30 13:02:55.874724 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:55.874677 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-istiod-ca-cert\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" Oct 30 13:02:55.874724 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:55.874698 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-istio-data\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" Oct 30 13:02:55.874925 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:55.874766 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-workload-certs\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" Oct 30 13:02:55.874925 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:55.874909 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-workload-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" Oct 30 13:02:55.875016 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:55.874921 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-credential-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" Oct 30 13:02:55.876742 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:55.876719 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-istio-envoy\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" Oct 30 13:02:55.877074 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:55.877055 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-istio-podinfo\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" Oct 30 13:02:55.899863 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:55.899776 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-istio-token\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" Oct 30 13:02:55.900019 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:55.899894 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk5wq\" (UniqueName: \"kubernetes.io/projected/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-kube-api-access-fk5wq\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" Oct 30 13:02:55.951402 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:55.951362 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" Oct 30 13:02:56.276234 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:56.276098 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj"] Oct 30 13:02:56.279196 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:02:56.279160 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5eb3b0b_b1d3_4b9e_83dc_a5179b6c6cb6.slice/crio-39bdf9894fa1bc5f53989243caee664bc894c26c6b9e19347691972cfb8e228e WatchSource:0}: Error finding container 39bdf9894fa1bc5f53989243caee664bc894c26c6b9e19347691972cfb8e228e: Status 404 returned error can't find the container with id 39bdf9894fa1bc5f53989243caee664bc894c26c6b9e19347691972cfb8e228e Oct 30 13:02:56.281572 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:56.281542 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:56.281651 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:56.281605 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:56.281651 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:56.281637 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:56.316015 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:56.315980 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" event={"ID":"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6","Type":"ContainerStarted","Data":"39bdf9894fa1bc5f53989243caee664bc894c26c6b9e19347691972cfb8e228e"} Oct 30 13:02:57.324353 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:57.324312 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" event={"ID":"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6","Type":"ContainerStarted","Data":"d017bb2161ab1ad8e3f37effa68a5f3c5f626dfaeecfba6342eddd0ebda6585e"} Oct 30 13:02:57.410564 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:57.410512 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" podStartSLOduration=2.410495765 podStartE2EDuration="2.410495765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:57.392001104 +0000 UTC m=+1075.995499553" watchObservedRunningTime="2026-10-30 13:02:57.410495765 +0000 UTC m=+1076.013994210" Oct 30 13:02:57.955292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:57.954591 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" Oct 30 13:02:57.959121 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:57.959091 4931 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 13:02:57.959319 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:57.959151 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" podUID="e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 13:02:58.953287 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:58.952553 4931 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 13:02:58.953287 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:58.952611 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" podUID="e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 13:02:59.953620 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:59.953581 4931 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 13:02:59.954201 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:02:59.954166 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" podUID="e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 13:03:00.187438 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:00.187397 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8"] Oct 30 13:03:00.192432 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:00.192410 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:03:00.223936 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:00.223906 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-dd90cb\"/\"distinct-2-w6-dd90cb-kserve-self-signed-certs\"" Oct 30 13:03:00.227702 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:00.227677 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8"] Oct 30 13:03:00.332046 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:00.332005 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ad1989a-c73f-4647-9e50-855605bdb08e-dshm\") pod \"distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8\" (UID: \"7ad1989a-c73f-4647-9e50-855605bdb08e\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:03:00.332250 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:00.332102 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ad1989a-c73f-4647-9e50-855605bdb08e-home\") pod \"distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8\" (UID: \"7ad1989a-c73f-4647-9e50-855605bdb08e\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:03:00.332250 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:00.332144 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ad1989a-c73f-4647-9e50-855605bdb08e-model-cache\") pod \"distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8\" (UID: \"7ad1989a-c73f-4647-9e50-855605bdb08e\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:03:00.332250 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:00.332201 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ad1989a-c73f-4647-9e50-855605bdb08e-tmp-dir\") pod \"distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8\" (UID: \"7ad1989a-c73f-4647-9e50-855605bdb08e\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:03:00.332451 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:00.332300 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhx74\" (UniqueName: \"kubernetes.io/projected/7ad1989a-c73f-4647-9e50-855605bdb08e-kube-api-access-nhx74\") pod \"distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8\" (UID: \"7ad1989a-c73f-4647-9e50-855605bdb08e\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:03:00.332451 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:00.332382 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ad1989a-c73f-4647-9e50-855605bdb08e-tls-certs\") pod \"distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8\" (UID: \"7ad1989a-c73f-4647-9e50-855605bdb08e\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:03:00.442087 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:00.442041 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ad1989a-c73f-4647-9e50-855605bdb08e-dshm\") pod \"distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8\" (UID: \"7ad1989a-c73f-4647-9e50-855605bdb08e\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:03:00.442874 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:00.442851 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ad1989a-c73f-4647-9e50-855605bdb08e-home\") pod \"distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8\" (UID: \"7ad1989a-c73f-4647-9e50-855605bdb08e\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:03:00.443468 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:00.443450 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ad1989a-c73f-4647-9e50-855605bdb08e-model-cache\") pod \"distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8\" (UID: \"7ad1989a-c73f-4647-9e50-855605bdb08e\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:03:00.443989 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:00.443860 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ad1989a-c73f-4647-9e50-855605bdb08e-model-cache\") pod \"distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8\" (UID: \"7ad1989a-c73f-4647-9e50-855605bdb08e\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:03:00.444125 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:00.443375 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ad1989a-c73f-4647-9e50-855605bdb08e-home\") pod \"distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8\" (UID: \"7ad1989a-c73f-4647-9e50-855605bdb08e\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:03:00.444326 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:00.444309 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ad1989a-c73f-4647-9e50-855605bdb08e-tmp-dir\") pod \"distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8\" (UID: \"7ad1989a-c73f-4647-9e50-855605bdb08e\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:03:00.444777 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:00.444760 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhx74\" (UniqueName: \"kubernetes.io/projected/7ad1989a-c73f-4647-9e50-855605bdb08e-kube-api-access-nhx74\") pod \"distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8\" (UID: \"7ad1989a-c73f-4647-9e50-855605bdb08e\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:03:00.445315 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:00.445294 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ad1989a-c73f-4647-9e50-855605bdb08e-tls-certs\") pod \"distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8\" (UID: \"7ad1989a-c73f-4647-9e50-855605bdb08e\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:03:00.446017 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:00.444701 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ad1989a-c73f-4647-9e50-855605bdb08e-tmp-dir\") pod \"distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8\" (UID: \"7ad1989a-c73f-4647-9e50-855605bdb08e\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:03:00.451934 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:00.451890 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ad1989a-c73f-4647-9e50-855605bdb08e-tls-certs\") pod \"distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8\" (UID: \"7ad1989a-c73f-4647-9e50-855605bdb08e\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:03:00.472002 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:00.471900 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ad1989a-c73f-4647-9e50-855605bdb08e-dshm\") pod \"distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8\" (UID: \"7ad1989a-c73f-4647-9e50-855605bdb08e\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:03:00.485062 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:00.484988 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhx74\" (UniqueName: \"kubernetes.io/projected/7ad1989a-c73f-4647-9e50-855605bdb08e-kube-api-access-nhx74\") pod \"distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8\" (UID: \"7ad1989a-c73f-4647-9e50-855605bdb08e\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:03:00.514524 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:00.513577 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:03:00.952660 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:00.952624 4931 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 13:03:00.957406 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:00.957364 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" podUID="e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 13:03:00.979212 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:00.979188 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8"] Oct 30 13:03:00.979345 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:03:00.979232 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ad1989a_c73f_4647_9e50_855605bdb08e.slice/crio-0b4be3edfc5ed83ef40be03912f53313fab8c3385241223c30d8218e297c38b5 WatchSource:0}: Error finding container 0b4be3edfc5ed83ef40be03912f53313fab8c3385241223c30d8218e297c38b5: Status 404 returned error can't find the container with id 0b4be3edfc5ed83ef40be03912f53313fab8c3385241223c30d8218e297c38b5 Oct 30 13:03:01.382832 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:01.382752 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" event={"ID":"7ad1989a-c73f-4647-9e50-855605bdb08e","Type":"ContainerStarted","Data":"0af1f8fcf29ba69dde70ce366df699a639fbe7aa39be5feb7aff4f8b3b3fb6d3"} Oct 30 13:03:01.382832 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:01.382792 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" event={"ID":"7ad1989a-c73f-4647-9e50-855605bdb08e","Type":"ContainerStarted","Data":"0b4be3edfc5ed83ef40be03912f53313fab8c3385241223c30d8218e297c38b5"} Oct 30 13:03:01.467303 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:01.466184 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" podStartSLOduration=1.466165293 podStartE2EDuration="1.466165293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:01.437635794 +0000 UTC m=+1080.041134256" watchObservedRunningTime="2026-10-30 13:03:01.466165293 +0000 UTC m=+1080.069663740" Oct 30 13:03:01.515366 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:01.515337 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:03:01.957291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:01.956564 4931 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 13:03:01.957291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:01.956620 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" podUID="e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 13:03:02.951983 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:02.951943 4931 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 13:03:02.952157 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:02.952020 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" podUID="e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 13:03:04.005346 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:04.005304 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" Oct 30 13:03:04.006315 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:04.006287 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" Oct 30 13:03:04.007309 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:04.007288 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" Oct 30 13:03:05.868624 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:05.868585 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc4896a9-25mnm"] Oct 30 13:03:05.872989 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:05.872962 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4896a9-25mnm" Oct 30 13:03:05.898200 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:05.898175 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:03:05.898384 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:05.898253 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:03:05.924208 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:05.924173 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59fe0cc8-1d0a-4b68-9111-070ffa1d4a08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc4896a9-25mnm\" (UID: \"59fe0cc8-1d0a-4b68-9111-070ffa1d4a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4896a9-25mnm" Oct 30 13:03:05.924445 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:05.924369 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59fe0cc8-1d0a-4b68-9111-070ffa1d4a08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc4896a9-25mnm\" (UID: \"59fe0cc8-1d0a-4b68-9111-070ffa1d4a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4896a9-25mnm" Oct 30 13:03:05.924445 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:05.924401 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwh5k\" (UniqueName: \"kubernetes.io/projected/59fe0cc8-1d0a-4b68-9111-070ffa1d4a08-kube-api-access-qwh5k\") pod \"maas-api-revoke-sub-fc4896a9-25mnm\" (UID: \"59fe0cc8-1d0a-4b68-9111-070ffa1d4a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4896a9-25mnm" Oct 30 13:03:05.951885 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:05.951849 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc4896a9-25mnm"] Oct 30 13:03:06.025421 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:06.025384 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59fe0cc8-1d0a-4b68-9111-070ffa1d4a08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc4896a9-25mnm\" (UID: \"59fe0cc8-1d0a-4b68-9111-070ffa1d4a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4896a9-25mnm" Oct 30 13:03:06.025617 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:06.025484 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59fe0cc8-1d0a-4b68-9111-070ffa1d4a08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc4896a9-25mnm\" (UID: \"59fe0cc8-1d0a-4b68-9111-070ffa1d4a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4896a9-25mnm" Oct 30 13:03:06.025617 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:06.025508 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwh5k\" (UniqueName: \"kubernetes.io/projected/59fe0cc8-1d0a-4b68-9111-070ffa1d4a08-kube-api-access-qwh5k\") pod \"maas-api-revoke-sub-fc4896a9-25mnm\" (UID: \"59fe0cc8-1d0a-4b68-9111-070ffa1d4a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4896a9-25mnm" Oct 30 13:03:06.026205 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:06.026178 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59fe0cc8-1d0a-4b68-9111-070ffa1d4a08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc4896a9-25mnm\" (UID: \"59fe0cc8-1d0a-4b68-9111-070ffa1d4a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4896a9-25mnm" Oct 30 13:03:06.027794 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:06.027776 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59fe0cc8-1d0a-4b68-9111-070ffa1d4a08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc4896a9-25mnm\" (UID: \"59fe0cc8-1d0a-4b68-9111-070ffa1d4a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4896a9-25mnm" Oct 30 13:03:06.057378 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:06.057348 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwh5k\" (UniqueName: \"kubernetes.io/projected/59fe0cc8-1d0a-4b68-9111-070ffa1d4a08-kube-api-access-qwh5k\") pod \"maas-api-revoke-sub-fc4896a9-25mnm\" (UID: \"59fe0cc8-1d0a-4b68-9111-070ffa1d4a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4896a9-25mnm" Oct 30 13:03:06.190068 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:06.189973 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4896a9-25mnm" Oct 30 13:03:06.497226 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:06.497193 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc4896a9-25mnm"] Oct 30 13:03:06.498218 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:03:06.498186 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59fe0cc8_1d0a_4b68_9111_070ffa1d4a08.slice/crio-44010a919696203898526242dba66ce0a00eadaef557ee8e5bf796a3b309a3ae WatchSource:0}: Error finding container 44010a919696203898526242dba66ce0a00eadaef557ee8e5bf796a3b309a3ae: Status 404 returned error can't find the container with id 44010a919696203898526242dba66ce0a00eadaef557ee8e5bf796a3b309a3ae Oct 30 13:03:07.351783 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:03:07.351741 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:03:07.423826 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:07.423786 4931 generic.go:356] "Generic (PLEG): container finished" podID="59fe0cc8-1d0a-4b68-9111-070ffa1d4a08" containerID="b49c78befa805615b8ae23633c69833dea77e4e1ef52f3f486334762422f32b7" exitCode=0 Oct 30 13:03:07.423964 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:07.423836 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4896a9-25mnm" event={"ID":"59fe0cc8-1d0a-4b68-9111-070ffa1d4a08","Type":"ContainerDied","Data":"b49c78befa805615b8ae23633c69833dea77e4e1ef52f3f486334762422f32b7"} Oct 30 13:03:07.423964 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:07.423922 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4896a9-25mnm" event={"ID":"59fe0cc8-1d0a-4b68-9111-070ffa1d4a08","Type":"ContainerStarted","Data":"44010a919696203898526242dba66ce0a00eadaef557ee8e5bf796a3b309a3ae"} Oct 30 13:03:08.434511 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:08.434237 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:03:08.601998 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:08.601971 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4896a9-25mnm" Oct 30 13:03:08.657105 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:08.657078 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59fe0cc8-1d0a-4b68-9111-070ffa1d4a08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59fe0cc8-1d0a-4b68-9111-070ffa1d4a08-maas-api-openshift-ca\") pod \"59fe0cc8-1d0a-4b68-9111-070ffa1d4a08\" (UID: \"59fe0cc8-1d0a-4b68-9111-070ffa1d4a08\") " Oct 30 13:03:08.657240 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:08.657155 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59fe0cc8-1d0a-4b68-9111-070ffa1d4a08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59fe0cc8-1d0a-4b68-9111-070ffa1d4a08-maas-api-xks-ca\") pod \"59fe0cc8-1d0a-4b68-9111-070ffa1d4a08\" (UID: \"59fe0cc8-1d0a-4b68-9111-070ffa1d4a08\") " Oct 30 13:03:08.657324 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:08.657316 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59fe0cc8-1d0a-4b68-9111-070ffa1d4a08-kube-api-access-qwh5k\" (UniqueName: \"kubernetes.io/projected/59fe0cc8-1d0a-4b68-9111-070ffa1d4a08-kube-api-access-qwh5k\") pod \"59fe0cc8-1d0a-4b68-9111-070ffa1d4a08\" (UID: \"59fe0cc8-1d0a-4b68-9111-070ffa1d4a08\") " Oct 30 13:03:08.657456 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:08.657431 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59fe0cc8-1d0a-4b68-9111-070ffa1d4a08-maas-api-openshift-ca" pod "59fe0cc8-1d0a-4b68-9111-070ffa1d4a08" (UID: "59fe0cc8-1d0a-4b68-9111-070ffa1d4a08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:03:08.657673 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:08.657656 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59fe0cc8-1d0a-4b68-9111-070ffa1d4a08-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:03:08.659203 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:08.659180 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59fe0cc8-1d0a-4b68-9111-070ffa1d4a08-maas-api-xks-ca" pod "59fe0cc8-1d0a-4b68-9111-070ffa1d4a08" (UID: "59fe0cc8-1d0a-4b68-9111-070ffa1d4a08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:03:08.659322 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:08.659303 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59fe0cc8-1d0a-4b68-9111-070ffa1d4a08-kube-api-access-qwh5k" pod "59fe0cc8-1d0a-4b68-9111-070ffa1d4a08" (UID: "59fe0cc8-1d0a-4b68-9111-070ffa1d4a08"). InnerVolumeSpecName "kube-api-access-qwh5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:03:08.759252 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:08.759221 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwh5k\" (UniqueName: \"kubernetes.io/projected/59fe0cc8-1d0a-4b68-9111-070ffa1d4a08-kube-api-access-qwh5k\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:03:08.759252 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:08.759249 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59fe0cc8-1d0a-4b68-9111-070ffa1d4a08-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:03:09.442200 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.442164 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4896a9-25mnm" Oct 30 13:03:09.442200 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.442193 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4896a9-25mnm" event={"ID":"59fe0cc8-1d0a-4b68-9111-070ffa1d4a08","Type":"ContainerDied","Data":"44010a919696203898526242dba66ce0a00eadaef557ee8e5bf796a3b309a3ae"} Oct 30 13:03:09.442815 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.442246 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44010a919696203898526242dba66ce0a00eadaef557ee8e5bf796a3b309a3ae" Oct 30 13:03:09.752276 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.752175 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b"] Oct 30 13:03:09.753029 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.753009 4931 cpu_manager.go:414] "containerMap: removing container" podUID="59fe0cc8-1d0a-4b68-9111-070ffa1d4a08" containerName="revoke-keys" Oct 30 13:03:09.753120 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.753031 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59fe0cc8-1d0a-4b68-9111-070ffa1d4a08" containerName="revoke-keys" Oct 30 13:03:09.753212 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.753194 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="59fe0cc8-1d0a-4b68-9111-070ffa1d4a08" containerName="revoke-keys" Oct 30 13:03:09.757634 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.757612 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" Oct 30 13:03:09.773568 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.773537 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e174882a-3e74-4174-99cf-dcf265fb86f6-istio-data\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" Oct 30 13:03:09.773759 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.773743 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e174882a-3e74-4174-99cf-dcf265fb86f6-istio-token\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" Oct 30 13:03:09.773897 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.773881 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e174882a-3e74-4174-99cf-dcf265fb86f6-credential-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" Oct 30 13:03:09.774073 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.774053 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c86qw\" (UniqueName: \"kubernetes.io/projected/e174882a-3e74-4174-99cf-dcf265fb86f6-kube-api-access-c86qw\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" Oct 30 13:03:09.774203 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.774186 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e174882a-3e74-4174-99cf-dcf265fb86f6-workload-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" Oct 30 13:03:09.774328 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.774310 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e174882a-3e74-4174-99cf-dcf265fb86f6-istio-envoy\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" Oct 30 13:03:09.774452 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.774437 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e174882a-3e74-4174-99cf-dcf265fb86f6-istio-podinfo\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" Oct 30 13:03:09.774569 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.774555 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e174882a-3e74-4174-99cf-dcf265fb86f6-workload-certs\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" Oct 30 13:03:09.774722 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.774708 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e174882a-3e74-4174-99cf-dcf265fb86f6-istiod-ca-cert\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" Oct 30 13:03:09.789613 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.789586 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b"] Oct 30 13:03:09.876414 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.876378 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e174882a-3e74-4174-99cf-dcf265fb86f6-istiod-ca-cert\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" Oct 30 13:03:09.876606 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.876476 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e174882a-3e74-4174-99cf-dcf265fb86f6-istio-data\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" Oct 30 13:03:09.876606 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.876537 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e174882a-3e74-4174-99cf-dcf265fb86f6-istio-token\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" Oct 30 13:03:09.876606 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.876591 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e174882a-3e74-4174-99cf-dcf265fb86f6-credential-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" Oct 30 13:03:09.876764 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.876672 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c86qw\" (UniqueName: \"kubernetes.io/projected/e174882a-3e74-4174-99cf-dcf265fb86f6-kube-api-access-c86qw\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" Oct 30 13:03:09.876764 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.876707 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e174882a-3e74-4174-99cf-dcf265fb86f6-workload-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" Oct 30 13:03:09.876764 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.876737 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e174882a-3e74-4174-99cf-dcf265fb86f6-istio-envoy\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" Oct 30 13:03:09.876914 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.876766 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e174882a-3e74-4174-99cf-dcf265fb86f6-istio-podinfo\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" Oct 30 13:03:09.876914 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.876800 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e174882a-3e74-4174-99cf-dcf265fb86f6-workload-certs\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" Oct 30 13:03:09.877200 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.877178 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e174882a-3e74-4174-99cf-dcf265fb86f6-workload-certs\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" Oct 30 13:03:09.877262 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.877194 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e174882a-3e74-4174-99cf-dcf265fb86f6-istiod-ca-cert\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" Oct 30 13:03:09.877463 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.877439 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e174882a-3e74-4174-99cf-dcf265fb86f6-istio-data\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" Oct 30 13:03:09.877895 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.877872 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e174882a-3e74-4174-99cf-dcf265fb86f6-workload-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" Oct 30 13:03:09.878034 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.877907 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e174882a-3e74-4174-99cf-dcf265fb86f6-credential-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" Oct 30 13:03:09.880786 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.880764 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e174882a-3e74-4174-99cf-dcf265fb86f6-istio-envoy\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" Oct 30 13:03:09.895332 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.895303 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e174882a-3e74-4174-99cf-dcf265fb86f6-istio-podinfo\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" Oct 30 13:03:09.908710 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.908679 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c86qw\" (UniqueName: \"kubernetes.io/projected/e174882a-3e74-4174-99cf-dcf265fb86f6-kube-api-access-c86qw\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" Oct 30 13:03:09.911101 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:09.911052 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e174882a-3e74-4174-99cf-dcf265fb86f6-istio-token\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" Oct 30 13:03:10.079534 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.079211 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" Oct 30 13:03:10.317340 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.315092 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft"] Oct 30 13:03:10.325726 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.325662 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" Oct 30 13:03:10.403970 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.402175 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83287de4-d8f9-4a84-96df-be7522695d0b-istio-token\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" Oct 30 13:03:10.403970 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.402231 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k42fv\" (UniqueName: \"kubernetes.io/projected/83287de4-d8f9-4a84-96df-be7522695d0b-kube-api-access-k42fv\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" Oct 30 13:03:10.403970 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.402302 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83287de4-d8f9-4a84-96df-be7522695d0b-workload-certs\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" Oct 30 13:03:10.403970 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.402400 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83287de4-d8f9-4a84-96df-be7522695d0b-istio-data\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" Oct 30 13:03:10.403970 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.402458 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83287de4-d8f9-4a84-96df-be7522695d0b-workload-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" Oct 30 13:03:10.403970 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.402493 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83287de4-d8f9-4a84-96df-be7522695d0b-istio-envoy\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" Oct 30 13:03:10.403970 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.402539 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83287de4-d8f9-4a84-96df-be7522695d0b-credential-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" Oct 30 13:03:10.403970 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.402591 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83287de4-d8f9-4a84-96df-be7522695d0b-istio-podinfo\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" Oct 30 13:03:10.403970 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.402626 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83287de4-d8f9-4a84-96df-be7522695d0b-istiod-ca-cert\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" Oct 30 13:03:10.592297 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.573965 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:03:10.611299 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.609996 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83287de4-d8f9-4a84-96df-be7522695d0b-istio-token\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" Oct 30 13:03:10.611299 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.610053 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k42fv\" (UniqueName: \"kubernetes.io/projected/83287de4-d8f9-4a84-96df-be7522695d0b-kube-api-access-k42fv\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" Oct 30 13:03:10.611299 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.610100 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83287de4-d8f9-4a84-96df-be7522695d0b-workload-certs\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" Oct 30 13:03:10.611299 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.610185 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83287de4-d8f9-4a84-96df-be7522695d0b-istio-data\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" Oct 30 13:03:10.611299 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.610283 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83287de4-d8f9-4a84-96df-be7522695d0b-workload-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" Oct 30 13:03:10.611299 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.610324 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83287de4-d8f9-4a84-96df-be7522695d0b-istio-envoy\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" Oct 30 13:03:10.611299 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.610364 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83287de4-d8f9-4a84-96df-be7522695d0b-credential-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" Oct 30 13:03:10.611299 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.610407 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83287de4-d8f9-4a84-96df-be7522695d0b-istio-podinfo\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" Oct 30 13:03:10.611299 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.610442 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83287de4-d8f9-4a84-96df-be7522695d0b-istiod-ca-cert\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" Oct 30 13:03:10.611918 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.611421 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83287de4-d8f9-4a84-96df-be7522695d0b-istiod-ca-cert\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" Oct 30 13:03:10.631968 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.612224 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83287de4-d8f9-4a84-96df-be7522695d0b-workload-certs\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" Oct 30 13:03:10.631968 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.612511 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83287de4-d8f9-4a84-96df-be7522695d0b-istio-data\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" Oct 30 13:03:10.631968 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.612751 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83287de4-d8f9-4a84-96df-be7522695d0b-workload-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" Oct 30 13:03:10.631968 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.622098 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83287de4-d8f9-4a84-96df-be7522695d0b-credential-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" Oct 30 13:03:10.820210 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.760256 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83287de4-d8f9-4a84-96df-be7522695d0b-istio-podinfo\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" Oct 30 13:03:10.820210 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.786587 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft"] Oct 30 13:03:10.896463 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.882756 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83287de4-d8f9-4a84-96df-be7522695d0b-istio-envoy\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" Oct 30 13:03:10.923547 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.919448 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:10.923547 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.919596 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:10.923547 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.919700 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:10.937473 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.936752 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83287de4-d8f9-4a84-96df-be7522695d0b-istio-token\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" Oct 30 13:03:10.958288 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.953308 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k42fv\" (UniqueName: \"kubernetes.io/projected/83287de4-d8f9-4a84-96df-be7522695d0b-kube-api-access-k42fv\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" Oct 30 13:03:10.983679 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:10.963458 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b"] Oct 30 13:03:11.020090 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:11.018491 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:03:11.339390 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:11.339044 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" Oct 30 13:03:11.668145 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:11.668059 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:03:12.106260 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:12.106222 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" event={"ID":"e174882a-3e74-4174-99cf-dcf265fb86f6","Type":"ContainerStarted","Data":"34bb6dd9415332a7ff06bd21408e100f0b433bfff710c7807d0bbe230e5daf94"} Oct 30 13:03:12.114222 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:12.114191 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" event={"ID":"e174882a-3e74-4174-99cf-dcf265fb86f6","Type":"ContainerStarted","Data":"25839ecf684dd1c16888c423eda7aab3af86efa181b52687b67f71531b71a133"} Oct 30 13:03:12.161743 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:12.161686 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" podStartSLOduration=3.161668564 podStartE2EDuration="3.161668564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:12.160064452 +0000 UTC m=+1090.763562927" watchObservedRunningTime="2026-10-30 13:03:12.161668564 +0000 UTC m=+1090.765167012" Oct 30 13:03:12.286303 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:12.269717 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft"] Oct 30 13:03:12.348039 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:12.346329 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:12.348361 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:12.348329 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:12.348561 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:12.348540 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:13.081906 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:13.081870 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" Oct 30 13:03:13.083683 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:13.083657 4931 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 13:03:13.083799 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:13.083713 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" podUID="e174882a-3e74-4174-99cf-dcf265fb86f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 13:03:13.117365 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:13.117324 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" event={"ID":"83287de4-d8f9-4a84-96df-be7522695d0b","Type":"ContainerStarted","Data":"a92616be2650f4c91ab2936612328448d7fc2ec46e0128967022d58879920352"} Oct 30 13:03:13.117669 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:13.117622 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" event={"ID":"83287de4-d8f9-4a84-96df-be7522695d0b","Type":"ContainerStarted","Data":"1559cf001b0f80d10256eb872d1ea6ce74681834ffc34fae45138450d3879c69"} Oct 30 13:03:13.182948 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:13.182891 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" podStartSLOduration=3.182870915 podStartE2EDuration="3.182870915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:13.180703886 +0000 UTC m=+1091.784202332" watchObservedRunningTime="2026-10-30 13:03:13.182870915 +0000 UTC m=+1091.786369364" Oct 30 13:03:13.368355 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:13.368081 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" Oct 30 13:03:13.374300 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:13.374178 4931 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:03:13.374300 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:13.374237 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" podUID="83287de4-d8f9-4a84-96df-be7522695d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 13:03:14.084293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:14.083628 4931 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 13:03:14.084293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:14.083688 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" podUID="e174882a-3e74-4174-99cf-dcf265fb86f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 13:03:14.376346 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:14.375713 4931 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:03:14.376346 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:14.375774 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" podUID="83287de4-d8f9-4a84-96df-be7522695d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 13:03:15.082071 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:15.081625 4931 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 13:03:15.082071 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:15.081690 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" podUID="e174882a-3e74-4174-99cf-dcf265fb86f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 13:03:15.388345 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:15.375911 4931 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:03:15.388345 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:15.375975 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" podUID="83287de4-d8f9-4a84-96df-be7522695d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 13:03:16.089601 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:16.088705 4931 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 13:03:16.089601 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:16.088765 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" podUID="e174882a-3e74-4174-99cf-dcf265fb86f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 13:03:16.373866 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:16.373769 4931 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:03:16.374152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:16.374105 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" podUID="83287de4-d8f9-4a84-96df-be7522695d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 13:03:17.083294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:17.082699 4931 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 13:03:17.083294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:17.082778 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" podUID="e174882a-3e74-4174-99cf-dcf265fb86f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 13:03:17.369719 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:17.369622 4931 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:03:17.370016 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:17.369982 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" podUID="83287de4-d8f9-4a84-96df-be7522695d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 13:03:18.081450 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:18.081414 4931 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 13:03:18.081631 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:18.081479 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" podUID="e174882a-3e74-4174-99cf-dcf265fb86f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 13:03:18.368094 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:18.367989 4931 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:03:18.368094 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:18.368078 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" podUID="83287de4-d8f9-4a84-96df-be7522695d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 13:03:19.368465 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:19.368422 4931 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:03:19.368932 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:19.368490 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" podUID="83287de4-d8f9-4a84-96df-be7522695d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 13:03:19.565905 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:19.565866 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" Oct 30 13:03:19.567397 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:19.567370 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" Oct 30 13:03:19.568257 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:19.568239 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" Oct 30 13:03:20.371764 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:20.371730 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" Oct 30 13:03:20.372209 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:20.372190 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" Oct 30 13:03:20.372882 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:20.372861 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" Oct 30 13:03:24.800508 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:24.800472 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw"] Oct 30 13:03:24.804795 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:24.804777 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" Oct 30 13:03:24.830625 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:24.830594 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw"] Oct 30 13:03:24.885179 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:24.885143 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-credential-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" Oct 30 13:03:24.885179 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:24.885184 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7swjc\" (UniqueName: \"kubernetes.io/projected/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-kube-api-access-7swjc\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" Oct 30 13:03:24.885413 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:24.885283 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-workload-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" Oct 30 13:03:24.885413 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:24.885341 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-istio-podinfo\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" Oct 30 13:03:24.885413 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:24.885402 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-istio-envoy\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" Oct 30 13:03:24.885540 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:24.885426 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-istio-token\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" Oct 30 13:03:24.885590 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:24.885568 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-istiod-ca-cert\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" Oct 30 13:03:24.885663 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:24.885646 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-istio-data\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" Oct 30 13:03:24.885718 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:24.885677 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-workload-certs\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" Oct 30 13:03:24.987015 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:24.986985 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-workload-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" Oct 30 13:03:24.987203 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:24.987042 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-istio-podinfo\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" Oct 30 13:03:24.987203 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:24.987120 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-istio-envoy\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" Oct 30 13:03:24.987203 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:24.987160 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-istio-token\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" Oct 30 13:03:24.987394 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:24.987224 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-istiod-ca-cert\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" Oct 30 13:03:24.987394 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:24.987326 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-istio-data\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" Oct 30 13:03:24.987503 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:24.987394 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-workload-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" Oct 30 13:03:24.987503 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:24.987450 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-workload-certs\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" Oct 30 13:03:24.987615 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:24.987507 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-credential-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" Oct 30 13:03:24.987615 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:24.987542 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7swjc\" (UniqueName: \"kubernetes.io/projected/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-kube-api-access-7swjc\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" Oct 30 13:03:24.987721 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:24.987684 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-istio-data\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" Oct 30 13:03:24.987788 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:24.987766 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-workload-certs\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" Oct 30 13:03:24.987892 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:24.987867 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-credential-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" Oct 30 13:03:24.988026 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:24.988011 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-istiod-ca-cert\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" Oct 30 13:03:24.989959 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:24.989934 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-istio-envoy\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" Oct 30 13:03:24.990130 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:24.990115 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-istio-podinfo\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" Oct 30 13:03:25.020551 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.020521 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7swjc\" (UniqueName: \"kubernetes.io/projected/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-kube-api-access-7swjc\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" Oct 30 13:03:25.020984 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.020967 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-istio-token\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" Oct 30 13:03:25.120119 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.120050 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" Oct 30 13:03:25.450022 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.449984 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw"] Oct 30 13:03:25.454683 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:03:25.454646 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e36d2e_bb23_46ba_bd2d_2f52fa9f4e63.slice/crio-696732f5cd5524c151e92574e42f8d7098af380568a2bd18359e211c9afc7fb8 WatchSource:0}: Error finding container 696732f5cd5524c151e92574e42f8d7098af380568a2bd18359e211c9afc7fb8: Status 404 returned error can't find the container with id 696732f5cd5524c151e92574e42f8d7098af380568a2bd18359e211c9afc7fb8 Oct 30 13:03:25.458003 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.457970 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:25.458132 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.458035 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:25.458132 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.458070 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:25.571772 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.571736 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l"] Oct 30 13:03:25.577042 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.577009 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" Oct 30 13:03:25.616461 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.616428 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l"] Oct 30 13:03:25.704343 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.703740 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af73e007-df5a-4dcb-9116-1a8764d29f7d-workload-certs\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" Oct 30 13:03:25.704343 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.703841 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af73e007-df5a-4dcb-9116-1a8764d29f7d-istio-token\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" Oct 30 13:03:25.704343 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.703876 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af73e007-df5a-4dcb-9116-1a8764d29f7d-istio-envoy\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" Oct 30 13:03:25.704343 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.703970 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af73e007-df5a-4dcb-9116-1a8764d29f7d-credential-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" Oct 30 13:03:25.704343 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.704006 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af73e007-df5a-4dcb-9116-1a8764d29f7d-workload-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" Oct 30 13:03:25.704343 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.704044 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af73e007-df5a-4dcb-9116-1a8764d29f7d-istio-data\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" Oct 30 13:03:25.704343 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.704070 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwkdd\" (UniqueName: \"kubernetes.io/projected/af73e007-df5a-4dcb-9116-1a8764d29f7d-kube-api-access-dwkdd\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" Oct 30 13:03:25.704343 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.704143 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af73e007-df5a-4dcb-9116-1a8764d29f7d-istio-podinfo\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" Oct 30 13:03:25.704343 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.704220 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af73e007-df5a-4dcb-9116-1a8764d29f7d-istiod-ca-cert\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" Oct 30 13:03:25.805650 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.805602 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af73e007-df5a-4dcb-9116-1a8764d29f7d-credential-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" Oct 30 13:03:25.806175 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.805662 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af73e007-df5a-4dcb-9116-1a8764d29f7d-workload-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" Oct 30 13:03:25.806175 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.805706 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af73e007-df5a-4dcb-9116-1a8764d29f7d-istio-data\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" Oct 30 13:03:25.806175 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.805735 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwkdd\" (UniqueName: \"kubernetes.io/projected/af73e007-df5a-4dcb-9116-1a8764d29f7d-kube-api-access-dwkdd\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" Oct 30 13:03:25.806175 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.805807 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af73e007-df5a-4dcb-9116-1a8764d29f7d-istio-podinfo\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" Oct 30 13:03:25.806175 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.805892 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af73e007-df5a-4dcb-9116-1a8764d29f7d-istiod-ca-cert\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" Oct 30 13:03:25.806175 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.805952 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af73e007-df5a-4dcb-9116-1a8764d29f7d-workload-certs\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" Oct 30 13:03:25.806175 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.806039 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af73e007-df5a-4dcb-9116-1a8764d29f7d-istio-token\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" Oct 30 13:03:25.806175 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.806070 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af73e007-df5a-4dcb-9116-1a8764d29f7d-istio-envoy\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" Oct 30 13:03:25.806175 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.806127 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af73e007-df5a-4dcb-9116-1a8764d29f7d-credential-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" Oct 30 13:03:25.806726 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.806453 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af73e007-df5a-4dcb-9116-1a8764d29f7d-istio-data\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" Oct 30 13:03:25.806726 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.806500 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af73e007-df5a-4dcb-9116-1a8764d29f7d-workload-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" Oct 30 13:03:25.807462 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.807434 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af73e007-df5a-4dcb-9116-1a8764d29f7d-workload-certs\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" Oct 30 13:03:25.807613 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.807587 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af73e007-df5a-4dcb-9116-1a8764d29f7d-istiod-ca-cert\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" Oct 30 13:03:25.809608 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.809580 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af73e007-df5a-4dcb-9116-1a8764d29f7d-istio-envoy\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" Oct 30 13:03:25.811098 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.811070 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af73e007-df5a-4dcb-9116-1a8764d29f7d-istio-podinfo\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" Oct 30 13:03:25.832338 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.832299 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af73e007-df5a-4dcb-9116-1a8764d29f7d-istio-token\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" Oct 30 13:03:25.832860 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.832830 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwkdd\" (UniqueName: \"kubernetes.io/projected/af73e007-df5a-4dcb-9116-1a8764d29f7d-kube-api-access-dwkdd\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" Oct 30 13:03:25.901835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:25.901793 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" Oct 30 13:03:26.035895 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:26.035373 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-946d7aa1-v7flq"] Oct 30 13:03:26.052716 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:26.052683 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-946d7aa1-v7flq" Oct 30 13:03:26.054144 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:26.054108 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-946d7aa1-v7flq"] Oct 30 13:03:26.085210 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:26.083692 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:03:26.085210 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:26.083842 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:03:26.245417 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:26.245379 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58b1995b-761f-4aad-9de2-2c4910eed9f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-946d7aa1-v7flq\" (UID: \"58b1995b-761f-4aad-9de2-2c4910eed9f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-946d7aa1-v7flq" Oct 30 13:03:26.245762 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:26.245737 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99tqc\" (UniqueName: \"kubernetes.io/projected/58b1995b-761f-4aad-9de2-2c4910eed9f2-kube-api-access-99tqc\") pod \"maas-api-revoke-sub-946d7aa1-v7flq\" (UID: \"58b1995b-761f-4aad-9de2-2c4910eed9f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-946d7aa1-v7flq" Oct 30 13:03:26.246148 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:26.246123 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58b1995b-761f-4aad-9de2-2c4910eed9f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-946d7aa1-v7flq\" (UID: \"58b1995b-761f-4aad-9de2-2c4910eed9f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-946d7aa1-v7flq" Oct 30 13:03:26.416434 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:26.378898 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58b1995b-761f-4aad-9de2-2c4910eed9f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-946d7aa1-v7flq\" (UID: \"58b1995b-761f-4aad-9de2-2c4910eed9f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-946d7aa1-v7flq" Oct 30 13:03:26.416434 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:26.378966 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99tqc\" (UniqueName: \"kubernetes.io/projected/58b1995b-761f-4aad-9de2-2c4910eed9f2-kube-api-access-99tqc\") pod \"maas-api-revoke-sub-946d7aa1-v7flq\" (UID: \"58b1995b-761f-4aad-9de2-2c4910eed9f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-946d7aa1-v7flq" Oct 30 13:03:26.416434 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:26.379142 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58b1995b-761f-4aad-9de2-2c4910eed9f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-946d7aa1-v7flq\" (UID: \"58b1995b-761f-4aad-9de2-2c4910eed9f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-946d7aa1-v7flq" Oct 30 13:03:26.419718 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:26.419687 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58b1995b-761f-4aad-9de2-2c4910eed9f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-946d7aa1-v7flq\" (UID: \"58b1995b-761f-4aad-9de2-2c4910eed9f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-946d7aa1-v7flq" Oct 30 13:03:26.424018 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:26.423986 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58b1995b-761f-4aad-9de2-2c4910eed9f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-946d7aa1-v7flq\" (UID: \"58b1995b-761f-4aad-9de2-2c4910eed9f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-946d7aa1-v7flq" Oct 30 13:03:26.427382 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:26.427215 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" event={"ID":"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63","Type":"ContainerStarted","Data":"550ee4479e76ad0efb2952086dd83b1277d42eb66ab1adc0c5d524053170dfb1"} Oct 30 13:03:26.427567 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:26.427546 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" event={"ID":"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63","Type":"ContainerStarted","Data":"696732f5cd5524c151e92574e42f8d7098af380568a2bd18359e211c9afc7fb8"} Oct 30 13:03:26.527344 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:26.521254 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99tqc\" (UniqueName: \"kubernetes.io/projected/58b1995b-761f-4aad-9de2-2c4910eed9f2-kube-api-access-99tqc\") pod \"maas-api-revoke-sub-946d7aa1-v7flq\" (UID: \"58b1995b-761f-4aad-9de2-2c4910eed9f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-946d7aa1-v7flq" Oct 30 13:03:26.528009 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:26.527946 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" podStartSLOduration=2.527925889 podStartE2EDuration="2.527925889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:26.524599153 +0000 UTC m=+1105.128097599" watchObservedRunningTime="2026-10-30 13:03:26.527925889 +0000 UTC m=+1105.131424338" Oct 30 13:03:26.595322 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:26.586246 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l"] Oct 30 13:03:26.657515 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:26.654736 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:26.657515 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:26.654830 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:26.657515 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:26.654876 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:26.710322 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:26.705744 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-946d7aa1-v7flq" Oct 30 13:03:27.125341 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:27.124454 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" Oct 30 13:03:27.125341 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:27.124658 4931 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 13:03:27.125341 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:27.124725 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" podUID="a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 13:03:27.493735 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:27.493693 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" event={"ID":"af73e007-df5a-4dcb-9116-1a8764d29f7d","Type":"ContainerStarted","Data":"fb0246045bc84fc2c4a4db0357308ca170af2cdb439fda77193422354b58ddaa"} Oct 30 13:03:27.494003 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:27.493981 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" event={"ID":"af73e007-df5a-4dcb-9116-1a8764d29f7d","Type":"ContainerStarted","Data":"0c4962d8d8772f7bc835059ba11d9d6b971b0ee0b0e6afb708482fdf513d3fc5"} Oct 30 13:03:27.578291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:27.556285 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" podUID="7ad1989a-c73f-4647-9e50-855605bdb08e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 13:03:27.625296 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:27.623347 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" podUID="606b252b-881c-4d40-8ba9-fc4ed937a995" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:27.687294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:27.686043 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" podStartSLOduration=2.6860250260000003 podStartE2EDuration="2.686025026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:27.599657558 +0000 UTC m=+1106.203156005" watchObservedRunningTime="2026-10-30 13:03:27.686025026 +0000 UTC m=+1106.289523473" Oct 30 13:03:27.721078 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:27.721035 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-946d7aa1-v7flq"] Oct 30 13:03:27.902551 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:27.902515 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" Oct 30 13:03:28.121185 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:28.121103 4931 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 13:03:28.121185 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:28.121154 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" podUID="a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 13:03:28.492639 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:28.492600 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-946d7aa1-v7flq" event={"ID":"58b1995b-761f-4aad-9de2-2c4910eed9f2","Type":"ContainerStarted","Data":"f8b7712629e498b801f0af97b15ba0a5716ca190762cffddde09a17a3c2f3a85"} Oct 30 13:03:28.493147 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:28.492660 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-946d7aa1-v7flq" event={"ID":"58b1995b-761f-4aad-9de2-2c4910eed9f2","Type":"ContainerStarted","Data":"fd4be88772ff8f1bd2ee1fdb7814f67246652ea8febebb7eb5963ce2df2190e8"} Oct 30 13:03:28.902880 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:28.902849 4931 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 13:03:28.903009 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:28.902904 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" podUID="af73e007-df5a-4dcb-9116-1a8764d29f7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 13:03:29.121043 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:29.121002 4931 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 13:03:29.121238 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:29.121065 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" podUID="a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 13:03:29.501521 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:29.501483 4931 generic.go:356] "Generic (PLEG): container finished" podID="58b1995b-761f-4aad-9de2-2c4910eed9f2" containerID="f8b7712629e498b801f0af97b15ba0a5716ca190762cffddde09a17a3c2f3a85" exitCode=0 Oct 30 13:03:29.501943 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:29.501546 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-946d7aa1-v7flq" event={"ID":"58b1995b-761f-4aad-9de2-2c4910eed9f2","Type":"ContainerDied","Data":"f8b7712629e498b801f0af97b15ba0a5716ca190762cffddde09a17a3c2f3a85"} Oct 30 13:03:29.902627 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:29.902552 4931 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 13:03:29.902627 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:29.902606 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" podUID="af73e007-df5a-4dcb-9116-1a8764d29f7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 13:03:30.120989 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:30.120953 4931 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 13:03:30.121170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:30.121009 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" podUID="a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 13:03:30.826078 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:30.826053 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-946d7aa1-v7flq" Oct 30 13:03:30.902637 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:30.902600 4931 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 13:03:30.902785 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:30.902661 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" podUID="af73e007-df5a-4dcb-9116-1a8764d29f7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 13:03:30.918177 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:30.918144 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58b1995b-761f-4aad-9de2-2c4910eed9f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58b1995b-761f-4aad-9de2-2c4910eed9f2-maas-api-openshift-ca\") pod \"58b1995b-761f-4aad-9de2-2c4910eed9f2\" (UID: \"58b1995b-761f-4aad-9de2-2c4910eed9f2\") " Oct 30 13:03:30.918366 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:30.918293 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58b1995b-761f-4aad-9de2-2c4910eed9f2-kube-api-access-99tqc\" (UniqueName: \"kubernetes.io/projected/58b1995b-761f-4aad-9de2-2c4910eed9f2-kube-api-access-99tqc\") pod \"58b1995b-761f-4aad-9de2-2c4910eed9f2\" (UID: \"58b1995b-761f-4aad-9de2-2c4910eed9f2\") " Oct 30 13:03:30.918366 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:30.918337 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58b1995b-761f-4aad-9de2-2c4910eed9f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58b1995b-761f-4aad-9de2-2c4910eed9f2-maas-api-xks-ca\") pod \"58b1995b-761f-4aad-9de2-2c4910eed9f2\" (UID: \"58b1995b-761f-4aad-9de2-2c4910eed9f2\") " Oct 30 13:03:30.918608 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:30.918582 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58b1995b-761f-4aad-9de2-2c4910eed9f2-maas-api-openshift-ca" pod "58b1995b-761f-4aad-9de2-2c4910eed9f2" (UID: "58b1995b-761f-4aad-9de2-2c4910eed9f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:03:30.918862 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:30.918844 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58b1995b-761f-4aad-9de2-2c4910eed9f2-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:03:30.920400 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:30.920377 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58b1995b-761f-4aad-9de2-2c4910eed9f2-maas-api-xks-ca" pod "58b1995b-761f-4aad-9de2-2c4910eed9f2" (UID: "58b1995b-761f-4aad-9de2-2c4910eed9f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:03:30.920849 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:30.920825 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58b1995b-761f-4aad-9de2-2c4910eed9f2-kube-api-access-99tqc" pod "58b1995b-761f-4aad-9de2-2c4910eed9f2" (UID: "58b1995b-761f-4aad-9de2-2c4910eed9f2"). InnerVolumeSpecName "kube-api-access-99tqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:03:31.020171 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:31.020134 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99tqc\" (UniqueName: \"kubernetes.io/projected/58b1995b-761f-4aad-9de2-2c4910eed9f2-kube-api-access-99tqc\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:03:31.020171 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:31.020168 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58b1995b-761f-4aad-9de2-2c4910eed9f2-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:03:31.121097 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:31.120994 4931 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 13:03:31.121097 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:31.121060 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" podUID="a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 13:03:31.523297 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:31.520035 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-946d7aa1-v7flq" Oct 30 13:03:31.523297 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:31.520048 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-946d7aa1-v7flq" event={"ID":"58b1995b-761f-4aad-9de2-2c4910eed9f2","Type":"ContainerDied","Data":"fd4be88772ff8f1bd2ee1fdb7814f67246652ea8febebb7eb5963ce2df2190e8"} Oct 30 13:03:31.523297 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:31.520101 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd4be88772ff8f1bd2ee1fdb7814f67246652ea8febebb7eb5963ce2df2190e8" Oct 30 13:03:31.903635 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:31.903534 4931 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 13:03:31.903635 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:31.903606 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" podUID="af73e007-df5a-4dcb-9116-1a8764d29f7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 13:03:32.120515 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:32.120476 4931 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 13:03:32.120721 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:32.120539 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" podUID="a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 13:03:32.902534 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:32.902500 4931 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 13:03:32.902720 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:32.902552 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" podUID="af73e007-df5a-4dcb-9116-1a8764d29f7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 13:03:33.120898 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:33.120861 4931 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 13:03:33.121392 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:33.120912 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" podUID="a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 13:03:33.902797 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:33.902745 4931 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 13:03:33.903021 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:33.902818 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" podUID="af73e007-df5a-4dcb-9116-1a8764d29f7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 13:03:34.125426 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:34.125395 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" Oct 30 13:03:34.125873 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:34.125766 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" Oct 30 13:03:34.126678 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:34.126658 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" Oct 30 13:03:35.257960 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:35.257924 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" Oct 30 13:03:35.259106 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:35.259080 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" Oct 30 13:03:35.259931 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:35.259912 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" Oct 30 13:03:37.351859 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:03:37.351826 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:03:38.139258 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:38.139227 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cdf5ad1-l28dq"] Oct 30 13:03:38.139818 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:38.139803 4931 cpu_manager.go:414] "containerMap: removing container" podUID="58b1995b-761f-4aad-9de2-2c4910eed9f2" containerName="revoke-keys" Oct 30 13:03:38.139869 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:38.139820 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58b1995b-761f-4aad-9de2-2c4910eed9f2" containerName="revoke-keys" Oct 30 13:03:38.139931 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:38.139922 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="58b1995b-761f-4aad-9de2-2c4910eed9f2" containerName="revoke-keys" Oct 30 13:03:38.144672 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:38.144652 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cdf5ad1-l28dq" Oct 30 13:03:38.163907 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:38.163880 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cdf5ad1-l28dq"] Oct 30 13:03:38.172423 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:38.172395 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:03:38.172561 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:38.172443 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:03:38.241079 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:38.241050 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqm56\" (UniqueName: \"kubernetes.io/projected/b5e5d14e-a36e-46c3-b550-adf7f3a9ad64-kube-api-access-zqm56\") pod \"maas-api-revoke-sub-5cdf5ad1-l28dq\" (UID: \"b5e5d14e-a36e-46c3-b550-adf7f3a9ad64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cdf5ad1-l28dq" Oct 30 13:03:38.241215 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:38.241116 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e5d14e-a36e-46c3-b550-adf7f3a9ad64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cdf5ad1-l28dq\" (UID: \"b5e5d14e-a36e-46c3-b550-adf7f3a9ad64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cdf5ad1-l28dq" Oct 30 13:03:38.241256 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:38.241234 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e5d14e-a36e-46c3-b550-adf7f3a9ad64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cdf5ad1-l28dq\" (UID: \"b5e5d14e-a36e-46c3-b550-adf7f3a9ad64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cdf5ad1-l28dq" Oct 30 13:03:38.342134 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:38.342102 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqm56\" (UniqueName: \"kubernetes.io/projected/b5e5d14e-a36e-46c3-b550-adf7f3a9ad64-kube-api-access-zqm56\") pod \"maas-api-revoke-sub-5cdf5ad1-l28dq\" (UID: \"b5e5d14e-a36e-46c3-b550-adf7f3a9ad64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cdf5ad1-l28dq" Oct 30 13:03:38.342293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:38.342177 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e5d14e-a36e-46c3-b550-adf7f3a9ad64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cdf5ad1-l28dq\" (UID: \"b5e5d14e-a36e-46c3-b550-adf7f3a9ad64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cdf5ad1-l28dq" Oct 30 13:03:38.342293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:38.342235 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e5d14e-a36e-46c3-b550-adf7f3a9ad64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cdf5ad1-l28dq\" (UID: \"b5e5d14e-a36e-46c3-b550-adf7f3a9ad64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cdf5ad1-l28dq" Oct 30 13:03:38.342812 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:38.342791 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e5d14e-a36e-46c3-b550-adf7f3a9ad64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cdf5ad1-l28dq\" (UID: \"b5e5d14e-a36e-46c3-b550-adf7f3a9ad64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cdf5ad1-l28dq" Oct 30 13:03:38.344589 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:38.344561 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e5d14e-a36e-46c3-b550-adf7f3a9ad64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cdf5ad1-l28dq\" (UID: \"b5e5d14e-a36e-46c3-b550-adf7f3a9ad64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cdf5ad1-l28dq" Oct 30 13:03:38.366744 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:38.366721 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqm56\" (UniqueName: \"kubernetes.io/projected/b5e5d14e-a36e-46c3-b550-adf7f3a9ad64-kube-api-access-zqm56\") pod \"maas-api-revoke-sub-5cdf5ad1-l28dq\" (UID: \"b5e5d14e-a36e-46c3-b550-adf7f3a9ad64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cdf5ad1-l28dq" Oct 30 13:03:38.458418 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:38.458386 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cdf5ad1-l28dq" Oct 30 13:03:38.811342 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:38.811305 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cdf5ad1-l28dq"] Oct 30 13:03:38.817491 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:03:38.817462 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5e5d14e_a36e_46c3_b550_adf7f3a9ad64.slice/crio-9b07a2699ea41c098cbe8e3a0759665b53ff920eae904e4d8163c016eb281897 WatchSource:0}: Error finding container 9b07a2699ea41c098cbe8e3a0759665b53ff920eae904e4d8163c016eb281897: Status 404 returned error can't find the container with id 9b07a2699ea41c098cbe8e3a0759665b53ff920eae904e4d8163c016eb281897 Oct 30 13:03:39.594688 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:39.594653 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cdf5ad1-l28dq" event={"ID":"b5e5d14e-a36e-46c3-b550-adf7f3a9ad64","Type":"ContainerStarted","Data":"680b0f28da7a2ca6b8768f69a50369ba702131b27f01c38c61dd50078f9e0db6"} Oct 30 13:03:39.595588 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:39.594708 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cdf5ad1-l28dq" event={"ID":"b5e5d14e-a36e-46c3-b550-adf7f3a9ad64","Type":"ContainerStarted","Data":"9b07a2699ea41c098cbe8e3a0759665b53ff920eae904e4d8163c016eb281897"} Oct 30 13:03:39.676220 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:39.676155 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cdf5ad1-l28dq" podStartSLOduration=1.6761367040000001 podStartE2EDuration="1.676136704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:39.673022629 +0000 UTC m=+1118.276521076" watchObservedRunningTime="2026-10-30 13:03:39.676136704 +0000 UTC m=+1118.279635153" Oct 30 13:03:40.602932 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:40.602896 4931 generic.go:356] "Generic (PLEG): container finished" podID="b5e5d14e-a36e-46c3-b550-adf7f3a9ad64" containerID="680b0f28da7a2ca6b8768f69a50369ba702131b27f01c38c61dd50078f9e0db6" exitCode=0 Oct 30 13:03:40.603391 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:40.602972 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cdf5ad1-l28dq" event={"ID":"b5e5d14e-a36e-46c3-b550-adf7f3a9ad64","Type":"ContainerDied","Data":"680b0f28da7a2ca6b8768f69a50369ba702131b27f01c38c61dd50078f9e0db6"} Oct 30 13:03:41.798149 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:41.798125 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cdf5ad1-l28dq" Oct 30 13:03:41.811638 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:41.811606 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5e5d14e-a36e-46c3-b550-adf7f3a9ad64-kube-api-access-zqm56\" (UniqueName: \"kubernetes.io/projected/b5e5d14e-a36e-46c3-b550-adf7f3a9ad64-kube-api-access-zqm56\") pod \"b5e5d14e-a36e-46c3-b550-adf7f3a9ad64\" (UID: \"b5e5d14e-a36e-46c3-b550-adf7f3a9ad64\") " Oct 30 13:03:41.811770 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:41.811658 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5e5d14e-a36e-46c3-b550-adf7f3a9ad64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e5d14e-a36e-46c3-b550-adf7f3a9ad64-maas-api-xks-ca\") pod \"b5e5d14e-a36e-46c3-b550-adf7f3a9ad64\" (UID: \"b5e5d14e-a36e-46c3-b550-adf7f3a9ad64\") " Oct 30 13:03:41.811770 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:41.811707 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5e5d14e-a36e-46c3-b550-adf7f3a9ad64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e5d14e-a36e-46c3-b550-adf7f3a9ad64-maas-api-openshift-ca\") pod \"b5e5d14e-a36e-46c3-b550-adf7f3a9ad64\" (UID: \"b5e5d14e-a36e-46c3-b550-adf7f3a9ad64\") " Oct 30 13:03:41.812132 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:41.812107 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5e5d14e-a36e-46c3-b550-adf7f3a9ad64-maas-api-openshift-ca" pod "b5e5d14e-a36e-46c3-b550-adf7f3a9ad64" (UID: "b5e5d14e-a36e-46c3-b550-adf7f3a9ad64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:03:41.812335 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:41.812319 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e5d14e-a36e-46c3-b550-adf7f3a9ad64-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:03:41.813703 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:41.813682 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5e5d14e-a36e-46c3-b550-adf7f3a9ad64-maas-api-xks-ca" pod "b5e5d14e-a36e-46c3-b550-adf7f3a9ad64" (UID: "b5e5d14e-a36e-46c3-b550-adf7f3a9ad64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:03:41.813703 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:41.813692 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5e5d14e-a36e-46c3-b550-adf7f3a9ad64-kube-api-access-zqm56" pod "b5e5d14e-a36e-46c3-b550-adf7f3a9ad64" (UID: "b5e5d14e-a36e-46c3-b550-adf7f3a9ad64"). InnerVolumeSpecName "kube-api-access-zqm56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:03:41.914016 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:41.913946 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqm56\" (UniqueName: \"kubernetes.io/projected/b5e5d14e-a36e-46c3-b550-adf7f3a9ad64-kube-api-access-zqm56\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:03:41.914016 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:41.913971 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e5d14e-a36e-46c3-b550-adf7f3a9ad64-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:03:42.618319 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:42.618285 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cdf5ad1-l28dq" Oct 30 13:03:42.618319 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:42.618303 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cdf5ad1-l28dq" event={"ID":"b5e5d14e-a36e-46c3-b550-adf7f3a9ad64","Type":"ContainerDied","Data":"9b07a2699ea41c098cbe8e3a0759665b53ff920eae904e4d8163c016eb281897"} Oct 30 13:03:42.618526 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:42.618354 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b07a2699ea41c098cbe8e3a0759665b53ff920eae904e4d8163c016eb281897" Oct 30 13:03:47.698828 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:47.698793 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:03:49.965324 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:49.944376 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" podUID="65799c14-8de4-4b4a-90df-7cd257e1e581" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:49.981290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:49.979355 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" podUID="e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:50.188291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:50.187361 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" podUID="9577f571-30cd-4043-a3e4-72bf95837552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:50.526399 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:50.526356 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" podUID="d345fdd5-407b-4162-a2cd-64a40068cb3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:50.577396 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:50.577349 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" podUID="5a504237-e5a3-491f-8f84-0419cf502f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:50.768289 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:50.759361 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" podUID="b1e86eea-4824-4c50-8ad3-2a15c3365c3d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:50.820349 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:50.813491 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" podUID="64468005-61ea-4f2a-b72c-9c30c6a4d686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:50.899293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:50.884441 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" podUID="089cb620-913d-47c7-b8f7-9b2298c194bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:50.997293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:50.994356 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" podUID="e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:50.997293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:50.995159 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:03:51.251608 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:51.251559 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" podUID="9577f571-30cd-4043-a3e4-72bf95837552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:51.276929 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:51.255344 4931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" podUID="64468005-61ea-4f2a-b72c-9c30c6a4d686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": context deadline exceeded" Oct 30 13:03:51.278338 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:51.277638 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:03:51.380291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:51.354519 4931 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:03:51.380291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:51.354605 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" podUID="af73e007-df5a-4dcb-9116-1a8764d29f7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:51.444453 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:51.444402 4931 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:03:51.444776 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:51.444730 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" podUID="83287de4-d8f9-4a84-96df-be7522695d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:51.526531 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:51.526429 4931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" podUID="7ad1989a-c73f-4647-9e50-855605bdb08e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:51.686806 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:51.526767 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" podUID="7ad1989a-c73f-4647-9e50-855605bdb08e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:51.712367 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:51.678342 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" podUID="606b252b-881c-4d40-8ba9-fc4ed937a995" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:51.713459 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:51.713421 4931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" podUID="9577f571-30cd-4043-a3e4-72bf95837552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:51.713720 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:51.713698 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" podUID="0c5a9f00-6d5b-4860-a210-2fec24afb322" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 13:03:51.745338 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:51.744599 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" podUID="5a504237-e5a3-491f-8f84-0419cf502f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:52.293292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:52.280749 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" podUID="65799c14-8de4-4b4a-90df-7cd257e1e581" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:52.298251 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:52.294441 4931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" podUID="65799c14-8de4-4b4a-90df-7cd257e1e581" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Oct 30 13:03:52.298251 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:52.294683 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" podUID="64468005-61ea-4f2a-b72c-9c30c6a4d686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 13:03:52.376552 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:52.376492 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" podUID="e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 13:03:52.399831 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:52.399353 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" podUID="9577f571-30cd-4043-a3e4-72bf95837552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 13:03:52.558496 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:52.558386 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" podUID="d345fdd5-407b-4162-a2cd-64a40068cb3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:52.735293 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:03:52.715975 4931 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.189s" Oct 30 13:03:52.770573 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:52.770542 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:03:52.770812 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:52.770796 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 13:03:52.793303 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:52.742155 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" podUID="0c5a9f00-6d5b-4860-a210-2fec24afb322" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:52.846376 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:52.756567 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" podUID="5a504237-e5a3-491f-8f84-0419cf502f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:52.847637 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:52.847608 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 13:03:52.848682 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:52.742206 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" podUID="7ad1989a-c73f-4647-9e50-855605bdb08e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:52.891292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:52.742248 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" podUID="606b252b-881c-4d40-8ba9-fc4ed937a995" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 13:03:52.891292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:52.793714 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" podUID="b1e86eea-4824-4c50-8ad3-2a15c3365c3d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:52.891292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:52.870670 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:03:52.891292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:52.870746 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 13:03:53.281486 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:53.281443 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" podUID="65799c14-8de4-4b4a-90df-7cd257e1e581" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:53.289294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:53.285974 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" podUID="089cb620-913d-47c7-b8f7-9b2298c194bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:53.327292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:53.298839 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:03:53.327292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:53.301240 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" podUID="64468005-61ea-4f2a-b72c-9c30c6a4d686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:53.472530 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:53.409576 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" podUID="9577f571-30cd-4043-a3e4-72bf95837552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:53.495564 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:53.413975 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" podUID="e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:53.808897 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:53.808843 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" podUID="d345fdd5-407b-4162-a2cd-64a40068cb3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:53.838863 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:53.838823 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 13:03:53.839055 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:53.838905 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 13:03:53.839055 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:53.838950 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 13:03:53.857669 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:53.857384 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" podUID="5a504237-e5a3-491f-8f84-0419cf502f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:53.908000 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:53.903232 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:03:54.114100 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:54.114020 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:03:54.222120 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:54.222087 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 13:03:54.280291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:54.280242 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 13:03:54.290111 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:54.290077 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:03:54.337287 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:54.337186 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:03:54.356498 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:54.356467 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:03:56.431519 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:56.431487 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0132418-n4459"] Oct 30 13:03:56.432054 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:56.432041 4931 cpu_manager.go:414] "containerMap: removing container" podUID="b5e5d14e-a36e-46c3-b550-adf7f3a9ad64" containerName="revoke-keys" Oct 30 13:03:56.432099 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:56.432056 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5e5d14e-a36e-46c3-b550-adf7f3a9ad64" containerName="revoke-keys" Oct 30 13:03:56.432157 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:56.432148 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5e5d14e-a36e-46c3-b550-adf7f3a9ad64" containerName="revoke-keys" Oct 30 13:03:56.435652 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:56.435628 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0132418-n4459" Oct 30 13:03:56.454996 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:56.454977 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:03:56.455320 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:56.455300 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:03:56.463076 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:56.463054 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d340149-7ccb-4395-a21d-55cdadff7b7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0132418-n4459\" (UID: \"8d340149-7ccb-4395-a21d-55cdadff7b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0132418-n4459" Oct 30 13:03:56.463167 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:56.463096 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d340149-7ccb-4395-a21d-55cdadff7b7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0132418-n4459\" (UID: \"8d340149-7ccb-4395-a21d-55cdadff7b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0132418-n4459" Oct 30 13:03:56.463208 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:56.463179 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prl9r\" (UniqueName: \"kubernetes.io/projected/8d340149-7ccb-4395-a21d-55cdadff7b7b-kube-api-access-prl9r\") pod \"maas-api-revoke-sub-a0132418-n4459\" (UID: \"8d340149-7ccb-4395-a21d-55cdadff7b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0132418-n4459" Oct 30 13:03:56.484048 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:56.484023 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0132418-n4459"] Oct 30 13:03:56.564225 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:56.564188 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d340149-7ccb-4395-a21d-55cdadff7b7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0132418-n4459\" (UID: \"8d340149-7ccb-4395-a21d-55cdadff7b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0132418-n4459" Oct 30 13:03:56.564446 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:56.564234 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d340149-7ccb-4395-a21d-55cdadff7b7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0132418-n4459\" (UID: \"8d340149-7ccb-4395-a21d-55cdadff7b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0132418-n4459" Oct 30 13:03:56.564446 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:56.564323 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prl9r\" (UniqueName: \"kubernetes.io/projected/8d340149-7ccb-4395-a21d-55cdadff7b7b-kube-api-access-prl9r\") pod \"maas-api-revoke-sub-a0132418-n4459\" (UID: \"8d340149-7ccb-4395-a21d-55cdadff7b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0132418-n4459" Oct 30 13:03:56.564953 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:56.564927 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d340149-7ccb-4395-a21d-55cdadff7b7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0132418-n4459\" (UID: \"8d340149-7ccb-4395-a21d-55cdadff7b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0132418-n4459" Oct 30 13:03:56.566623 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:56.566604 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d340149-7ccb-4395-a21d-55cdadff7b7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0132418-n4459\" (UID: \"8d340149-7ccb-4395-a21d-55cdadff7b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0132418-n4459" Oct 30 13:03:56.609774 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:56.609750 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prl9r\" (UniqueName: \"kubernetes.io/projected/8d340149-7ccb-4395-a21d-55cdadff7b7b-kube-api-access-prl9r\") pod \"maas-api-revoke-sub-a0132418-n4459\" (UID: \"8d340149-7ccb-4395-a21d-55cdadff7b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0132418-n4459" Oct 30 13:03:56.749374 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:56.749346 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0132418-n4459" Oct 30 13:03:57.057517 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:57.057480 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0132418-n4459"] Oct 30 13:03:57.138086 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:57.138048 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0132418-n4459" event={"ID":"8d340149-7ccb-4395-a21d-55cdadff7b7b","Type":"ContainerStarted","Data":"608a67b6eb1ca5e0ccb7b139d3548faaec48bc85d0c964c46254f770aacbec43"} Oct 30 13:03:58.147954 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:58.147923 4931 generic.go:356] "Generic (PLEG): container finished" podID="8d340149-7ccb-4395-a21d-55cdadff7b7b" containerID="3e167b960a71b02331e0ff88b5a79fa35851274c273284bedc45666b75fabc06" exitCode=0 Oct 30 13:03:58.148413 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:58.148012 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0132418-n4459" event={"ID":"8d340149-7ccb-4395-a21d-55cdadff7b7b","Type":"ContainerDied","Data":"3e167b960a71b02331e0ff88b5a79fa35851274c273284bedc45666b75fabc06"} Oct 30 13:03:59.346780 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:59.346751 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0132418-n4459" Oct 30 13:03:59.407165 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:59.407070 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d340149-7ccb-4395-a21d-55cdadff7b7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d340149-7ccb-4395-a21d-55cdadff7b7b-maas-api-xks-ca\") pod \"8d340149-7ccb-4395-a21d-55cdadff7b7b\" (UID: \"8d340149-7ccb-4395-a21d-55cdadff7b7b\") " Oct 30 13:03:59.407344 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:59.407163 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d340149-7ccb-4395-a21d-55cdadff7b7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d340149-7ccb-4395-a21d-55cdadff7b7b-maas-api-openshift-ca\") pod \"8d340149-7ccb-4395-a21d-55cdadff7b7b\" (UID: \"8d340149-7ccb-4395-a21d-55cdadff7b7b\") " Oct 30 13:03:59.407446 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:59.407424 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d340149-7ccb-4395-a21d-55cdadff7b7b-kube-api-access-prl9r\" (UniqueName: \"kubernetes.io/projected/8d340149-7ccb-4395-a21d-55cdadff7b7b-kube-api-access-prl9r\") pod \"8d340149-7ccb-4395-a21d-55cdadff7b7b\" (UID: \"8d340149-7ccb-4395-a21d-55cdadff7b7b\") " Oct 30 13:03:59.407615 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:59.407588 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d340149-7ccb-4395-a21d-55cdadff7b7b-maas-api-openshift-ca" pod "8d340149-7ccb-4395-a21d-55cdadff7b7b" (UID: "8d340149-7ccb-4395-a21d-55cdadff7b7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:03:59.407941 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:59.407925 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d340149-7ccb-4395-a21d-55cdadff7b7b-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:03:59.409871 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:59.409844 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d340149-7ccb-4395-a21d-55cdadff7b7b-maas-api-xks-ca" pod "8d340149-7ccb-4395-a21d-55cdadff7b7b" (UID: "8d340149-7ccb-4395-a21d-55cdadff7b7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:03:59.410147 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:59.410103 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d340149-7ccb-4395-a21d-55cdadff7b7b-kube-api-access-prl9r" pod "8d340149-7ccb-4395-a21d-55cdadff7b7b" (UID: "8d340149-7ccb-4395-a21d-55cdadff7b7b"). InnerVolumeSpecName "kube-api-access-prl9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:03:59.509246 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:59.509212 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prl9r\" (UniqueName: \"kubernetes.io/projected/8d340149-7ccb-4395-a21d-55cdadff7b7b-kube-api-access-prl9r\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:03:59.509246 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:03:59.509244 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d340149-7ccb-4395-a21d-55cdadff7b7b-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:04:00.173333 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:00.173291 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0132418-n4459" event={"ID":"8d340149-7ccb-4395-a21d-55cdadff7b7b","Type":"ContainerDied","Data":"608a67b6eb1ca5e0ccb7b139d3548faaec48bc85d0c964c46254f770aacbec43"} Oct 30 13:04:00.173505 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:00.173375 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="608a67b6eb1ca5e0ccb7b139d3548faaec48bc85d0c964c46254f770aacbec43" Oct 30 13:04:00.173505 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:00.173307 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0132418-n4459" Oct 30 13:04:07.353278 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:04:07.353226 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:04:11.667333 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:11.667282 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" podUID="0c5a9f00-6d5b-4860-a210-2fec24afb322" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:11.669721 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:11.668489 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" podUID="606b252b-881c-4d40-8ba9-fc4ed937a995" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:11.717686 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:11.717574 4931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" podUID="9577f571-30cd-4043-a3e4-72bf95837552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:11.759508 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:11.758367 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" podUID="b1e86eea-4824-4c50-8ad3-2a15c3365c3d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:11.862411 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:11.862360 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" podUID="5a504237-e5a3-491f-8f84-0419cf502f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:11.882400 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:11.882353 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" podUID="089cb620-913d-47c7-b8f7-9b2298c194bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:11.940214 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:11.940105 4931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" podUID="65799c14-8de4-4b4a-90df-7cd257e1e581" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:12.125492 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:12.125423 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" podUID="65799c14-8de4-4b4a-90df-7cd257e1e581" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:12.234786 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:12.233827 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" podUID="7ad1989a-c73f-4647-9e50-855605bdb08e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 13:04:12.318471 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:12.291310 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" podUID="d345fdd5-407b-4162-a2cd-64a40068cb3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:12.378289 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:12.377441 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" podUID="64468005-61ea-4f2a-b72c-9c30c6a4d686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 13:04:12.378606 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:12.378570 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" podUID="e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:12.415352 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:12.415294 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" podUID="9577f571-30cd-4043-a3e4-72bf95837552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:12.688889 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:12.686375 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" podUID="0c5a9f00-6d5b-4860-a210-2fec24afb322" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 13:04:12.688889 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:12.687168 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 13:04:12.728732 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:12.713792 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" podUID="606b252b-881c-4d40-8ba9-fc4ed937a995" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 13:04:12.729681 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:12.729640 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 13:04:12.780832 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:12.780787 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" podUID="b1e86eea-4824-4c50-8ad3-2a15c3365c3d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:12.835935 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:12.834974 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" Oct 30 13:04:12.918502 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:12.892448 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" podUID="5a504237-e5a3-491f-8f84-0419cf502f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:12.918502 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:12.915411 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:04:12.918502 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:12.917404 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" podUID="089cb620-913d-47c7-b8f7-9b2298c194bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:12.919806 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:12.918947 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 13:04:13.132572 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:13.132529 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" podUID="65799c14-8de4-4b4a-90df-7cd257e1e581" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 13:04:13.146864 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:13.146823 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:04:13.295292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:13.267226 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" podUID="7ad1989a-c73f-4647-9e50-855605bdb08e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:13.326203 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:13.296901 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" podUID="d345fdd5-407b-4162-a2cd-64a40068cb3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:13.328596 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:13.328559 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:04:13.338794 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:13.338770 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 13:04:13.412340 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:13.386968 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" podUID="e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:13.412340 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:13.397434 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:04:13.428815 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:13.428773 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" podUID="64468005-61ea-4f2a-b72c-9c30c6a4d686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:13.474805 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:13.430419 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" podUID="9577f571-30cd-4043-a3e4-72bf95837552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 13:04:13.476568 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:13.476543 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 13:04:13.476772 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:13.476757 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:04:13.703389 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:13.697396 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" podUID="0c5a9f00-6d5b-4860-a210-2fec24afb322" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:13.739409 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:13.731360 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" podUID="606b252b-881c-4d40-8ba9-fc4ed937a995" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 13:04:13.837610 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:13.837568 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" podUID="b1e86eea-4824-4c50-8ad3-2a15c3365c3d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 13:04:13.893177 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:13.893135 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" podUID="5a504237-e5a3-491f-8f84-0419cf502f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 13:04:13.924663 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:13.924616 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" podUID="089cb620-913d-47c7-b8f7-9b2298c194bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 13:04:14.137349 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:14.136394 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" podUID="65799c14-8de4-4b4a-90df-7cd257e1e581" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:14.332306 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:14.325079 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" podUID="7ad1989a-c73f-4647-9e50-855605bdb08e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:14.337312 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:14.332635 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" podUID="d345fdd5-407b-4162-a2cd-64a40068cb3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:14.395572 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:14.395467 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" podUID="e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:14.485420 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:14.485375 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" podUID="9577f571-30cd-4043-a3e4-72bf95837552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 13:04:14.486040 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:14.486007 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" podUID="64468005-61ea-4f2a-b72c-9c30c6a4d686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 13:04:14.712470 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:14.712414 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" podUID="0c5a9f00-6d5b-4860-a210-2fec24afb322" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:14.778408 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:14.745660 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" podUID="606b252b-881c-4d40-8ba9-fc4ed937a995" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:14.917549 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:14.917497 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" podUID="5a504237-e5a3-491f-8f84-0419cf502f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:14.983626 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:14.983518 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" podUID="b1e86eea-4824-4c50-8ad3-2a15c3365c3d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 13:04:15.030296 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:15.017872 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" podUID="089cb620-913d-47c7-b8f7-9b2298c194bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 13:04:15.460358 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:15.460317 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" podUID="d345fdd5-407b-4162-a2cd-64a40068cb3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:15.465465 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:15.462597 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" podUID="e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:15.472406 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:15.472380 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 13:04:15.473216 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:15.473184 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" podUID="65799c14-8de4-4b4a-90df-7cd257e1e581" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:15.474101 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:15.474077 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" podUID="7ad1989a-c73f-4647-9e50-855605bdb08e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 13:04:15.487309 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:15.486946 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 13:04:15.492601 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:15.487451 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:04:15.492601 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:15.487739 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" podUID="9577f571-30cd-4043-a3e4-72bf95837552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 13:04:15.493004 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:15.492941 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 13:04:15.498428 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:15.498401 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" podUID="64468005-61ea-4f2a-b72c-9c30c6a4d686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:15.501182 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:15.501161 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:04:15.502466 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:15.502410 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" Oct 30 13:04:15.518869 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:15.518845 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 13:04:15.518991 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:15.518902 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:04:15.518991 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:15.518934 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:04:15.518991 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:15.518964 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:04:15.523148 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:15.523124 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 13:04:20.395096 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:20.395058 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-183345e7-z2ghc"] Oct 30 13:04:20.395787 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:20.395769 4931 cpu_manager.go:414] "containerMap: removing container" podUID="8d340149-7ccb-4395-a21d-55cdadff7b7b" containerName="revoke-keys" Oct 30 13:04:20.395787 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:20.395786 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d340149-7ccb-4395-a21d-55cdadff7b7b" containerName="revoke-keys" Oct 30 13:04:20.395917 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:20.395896 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d340149-7ccb-4395-a21d-55cdadff7b7b" containerName="revoke-keys" Oct 30 13:04:20.399652 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:20.399635 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-183345e7-z2ghc" Oct 30 13:04:20.432922 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:20.432882 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvlw9\" (UniqueName: \"kubernetes.io/projected/784b2871-f3ab-4cdd-966d-be76d743fa2e-kube-api-access-jvlw9\") pod \"maas-api-revoke-sub-183345e7-z2ghc\" (UID: \"784b2871-f3ab-4cdd-966d-be76d743fa2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183345e7-z2ghc" Oct 30 13:04:20.433116 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:20.432953 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/784b2871-f3ab-4cdd-966d-be76d743fa2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-183345e7-z2ghc\" (UID: \"784b2871-f3ab-4cdd-966d-be76d743fa2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183345e7-z2ghc" Oct 30 13:04:20.433116 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:20.433087 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/784b2871-f3ab-4cdd-966d-be76d743fa2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-183345e7-z2ghc\" (UID: \"784b2871-f3ab-4cdd-966d-be76d743fa2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183345e7-z2ghc" Oct 30 13:04:20.434098 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:20.434071 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-183345e7-z2ghc"] Oct 30 13:04:20.441447 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:20.441419 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:04:20.441603 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:20.441465 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:04:20.534383 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:20.534344 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvlw9\" (UniqueName: \"kubernetes.io/projected/784b2871-f3ab-4cdd-966d-be76d743fa2e-kube-api-access-jvlw9\") pod \"maas-api-revoke-sub-183345e7-z2ghc\" (UID: \"784b2871-f3ab-4cdd-966d-be76d743fa2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183345e7-z2ghc" Oct 30 13:04:20.534575 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:20.534495 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/784b2871-f3ab-4cdd-966d-be76d743fa2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-183345e7-z2ghc\" (UID: \"784b2871-f3ab-4cdd-966d-be76d743fa2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183345e7-z2ghc" Oct 30 13:04:20.534657 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:20.534640 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/784b2871-f3ab-4cdd-966d-be76d743fa2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-183345e7-z2ghc\" (UID: \"784b2871-f3ab-4cdd-966d-be76d743fa2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183345e7-z2ghc" Oct 30 13:04:20.535327 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:20.535304 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/784b2871-f3ab-4cdd-966d-be76d743fa2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-183345e7-z2ghc\" (UID: \"784b2871-f3ab-4cdd-966d-be76d743fa2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183345e7-z2ghc" Oct 30 13:04:20.536922 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:20.536901 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/784b2871-f3ab-4cdd-966d-be76d743fa2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-183345e7-z2ghc\" (UID: \"784b2871-f3ab-4cdd-966d-be76d743fa2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183345e7-z2ghc" Oct 30 13:04:20.559201 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:20.559175 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvlw9\" (UniqueName: \"kubernetes.io/projected/784b2871-f3ab-4cdd-966d-be76d743fa2e-kube-api-access-jvlw9\") pod \"maas-api-revoke-sub-183345e7-z2ghc\" (UID: \"784b2871-f3ab-4cdd-966d-be76d743fa2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183345e7-z2ghc" Oct 30 13:04:20.713895 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:20.713860 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-183345e7-z2ghc" Oct 30 13:04:21.021573 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:04:21.021538 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784b2871_f3ab_4cdd_966d_be76d743fa2e.slice/crio-dba57968ed51332c2baf5e19ca697ba7da2b97656b5e70b6d82a5b46bf4357dc WatchSource:0}: Error finding container dba57968ed51332c2baf5e19ca697ba7da2b97656b5e70b6d82a5b46bf4357dc: Status 404 returned error can't find the container with id dba57968ed51332c2baf5e19ca697ba7da2b97656b5e70b6d82a5b46bf4357dc Oct 30 13:04:21.022218 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:21.022191 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-183345e7-z2ghc"] Oct 30 13:04:21.554761 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:21.554720 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-183345e7-z2ghc" event={"ID":"784b2871-f3ab-4cdd-966d-be76d743fa2e","Type":"ContainerStarted","Data":"b995d3ad578c5561c25b57890b0bea80c1f1ef95748500f6376751faa25e6274"} Oct 30 13:04:21.555152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:21.554782 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-183345e7-z2ghc" event={"ID":"784b2871-f3ab-4cdd-966d-be76d743fa2e","Type":"ContainerStarted","Data":"dba57968ed51332c2baf5e19ca697ba7da2b97656b5e70b6d82a5b46bf4357dc"} Oct 30 13:04:21.603919 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:21.603804 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-183345e7-z2ghc" podStartSLOduration=1.603786751 podStartE2EDuration="1.603786751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:21.600396779 +0000 UTC m=+1160.203895240" watchObservedRunningTime="2026-10-30 13:04:21.603786751 +0000 UTC m=+1160.207285205" Oct 30 13:04:22.563523 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:22.563488 4931 generic.go:356] "Generic (PLEG): container finished" podID="784b2871-f3ab-4cdd-966d-be76d743fa2e" containerID="b995d3ad578c5561c25b57890b0bea80c1f1ef95748500f6376751faa25e6274" exitCode=0 Oct 30 13:04:22.563946 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:22.563577 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-183345e7-z2ghc" event={"ID":"784b2871-f3ab-4cdd-966d-be76d743fa2e","Type":"ContainerDied","Data":"b995d3ad578c5561c25b57890b0bea80c1f1ef95748500f6376751faa25e6274"} Oct 30 13:04:23.763353 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:23.763315 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-183345e7-z2ghc" Oct 30 13:04:23.878460 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:23.878378 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/784b2871-f3ab-4cdd-966d-be76d743fa2e-kube-api-access-jvlw9\" (UniqueName: \"kubernetes.io/projected/784b2871-f3ab-4cdd-966d-be76d743fa2e-kube-api-access-jvlw9\") pod \"784b2871-f3ab-4cdd-966d-be76d743fa2e\" (UID: \"784b2871-f3ab-4cdd-966d-be76d743fa2e\") " Oct 30 13:04:23.878599 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:23.878539 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/784b2871-f3ab-4cdd-966d-be76d743fa2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/784b2871-f3ab-4cdd-966d-be76d743fa2e-maas-api-xks-ca\") pod \"784b2871-f3ab-4cdd-966d-be76d743fa2e\" (UID: \"784b2871-f3ab-4cdd-966d-be76d743fa2e\") " Oct 30 13:04:23.878636 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:23.878612 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/784b2871-f3ab-4cdd-966d-be76d743fa2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/784b2871-f3ab-4cdd-966d-be76d743fa2e-maas-api-openshift-ca\") pod \"784b2871-f3ab-4cdd-966d-be76d743fa2e\" (UID: \"784b2871-f3ab-4cdd-966d-be76d743fa2e\") " Oct 30 13:04:23.878961 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:23.878933 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/784b2871-f3ab-4cdd-966d-be76d743fa2e-maas-api-openshift-ca" pod "784b2871-f3ab-4cdd-966d-be76d743fa2e" (UID: "784b2871-f3ab-4cdd-966d-be76d743fa2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:23.880569 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:23.880545 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/784b2871-f3ab-4cdd-966d-be76d743fa2e-maas-api-xks-ca" pod "784b2871-f3ab-4cdd-966d-be76d743fa2e" (UID: "784b2871-f3ab-4cdd-966d-be76d743fa2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:23.880677 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:23.880662 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784b2871-f3ab-4cdd-966d-be76d743fa2e-kube-api-access-jvlw9" pod "784b2871-f3ab-4cdd-966d-be76d743fa2e" (UID: "784b2871-f3ab-4cdd-966d-be76d743fa2e"). InnerVolumeSpecName "kube-api-access-jvlw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:23.980637 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:23.980601 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/784b2871-f3ab-4cdd-966d-be76d743fa2e-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:04:23.980637 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:23.980633 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvlw9\" (UniqueName: \"kubernetes.io/projected/784b2871-f3ab-4cdd-966d-be76d743fa2e-kube-api-access-jvlw9\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:04:23.980853 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:23.980648 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/784b2871-f3ab-4cdd-966d-be76d743fa2e-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:04:24.294363 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:24.294328 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19c6ea11-sr6hx"] Oct 30 13:04:24.295044 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:24.295010 4931 cpu_manager.go:414] "containerMap: removing container" podUID="784b2871-f3ab-4cdd-966d-be76d743fa2e" containerName="revoke-keys" Oct 30 13:04:24.295044 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:24.295034 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="784b2871-f3ab-4cdd-966d-be76d743fa2e" containerName="revoke-keys" Oct 30 13:04:24.295218 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:24.295184 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="784b2871-f3ab-4cdd-966d-be76d743fa2e" containerName="revoke-keys" Oct 30 13:04:24.299166 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:24.299142 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c6ea11-sr6hx" Oct 30 13:04:24.323629 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:24.323593 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19c6ea11-sr6hx"] Oct 30 13:04:24.377292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:24.377236 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27871d28-f4r7w"] Oct 30 13:04:24.381562 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:24.381546 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27871d28-f4r7w" Oct 30 13:04:24.386580 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:24.386551 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5639c55b-e419-4b24-9a26-192e300660df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19c6ea11-sr6hx\" (UID: \"5639c55b-e419-4b24-9a26-192e300660df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c6ea11-sr6hx" Oct 30 13:04:24.386737 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:24.386717 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t67n\" (UniqueName: \"kubernetes.io/projected/5639c55b-e419-4b24-9a26-192e300660df-kube-api-access-2t67n\") pod \"maas-api-revoke-sub-19c6ea11-sr6hx\" (UID: \"5639c55b-e419-4b24-9a26-192e300660df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c6ea11-sr6hx" Oct 30 13:04:24.386930 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:24.386909 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5639c55b-e419-4b24-9a26-192e300660df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19c6ea11-sr6hx\" (UID: \"5639c55b-e419-4b24-9a26-192e300660df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c6ea11-sr6hx" Oct 30 13:04:24.407235 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:24.407207 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27871d28-f4r7w"] Oct 30 13:04:24.488817 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:24.488784 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5639c55b-e419-4b24-9a26-192e300660df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19c6ea11-sr6hx\" (UID: \"5639c55b-e419-4b24-9a26-192e300660df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c6ea11-sr6hx" Oct 30 13:04:24.489012 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:24.488839 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3463205-9bd3-431e-aa57-c6343263a1b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27871d28-f4r7w\" (UID: \"f3463205-9bd3-431e-aa57-c6343263a1b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27871d28-f4r7w" Oct 30 13:04:24.489012 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:24.488908 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5639c55b-e419-4b24-9a26-192e300660df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19c6ea11-sr6hx\" (UID: \"5639c55b-e419-4b24-9a26-192e300660df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c6ea11-sr6hx" Oct 30 13:04:24.489012 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:24.489004 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-797rl\" (UniqueName: \"kubernetes.io/projected/f3463205-9bd3-431e-aa57-c6343263a1b8-kube-api-access-797rl\") pod \"maas-api-revoke-sub-27871d28-f4r7w\" (UID: \"f3463205-9bd3-431e-aa57-c6343263a1b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27871d28-f4r7w" Oct 30 13:04:24.489190 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:24.489058 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t67n\" (UniqueName: \"kubernetes.io/projected/5639c55b-e419-4b24-9a26-192e300660df-kube-api-access-2t67n\") pod \"maas-api-revoke-sub-19c6ea11-sr6hx\" (UID: \"5639c55b-e419-4b24-9a26-192e300660df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c6ea11-sr6hx" Oct 30 13:04:24.489190 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:24.489137 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3463205-9bd3-431e-aa57-c6343263a1b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27871d28-f4r7w\" (UID: \"f3463205-9bd3-431e-aa57-c6343263a1b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27871d28-f4r7w" Oct 30 13:04:24.489546 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:24.489528 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5639c55b-e419-4b24-9a26-192e300660df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19c6ea11-sr6hx\" (UID: \"5639c55b-e419-4b24-9a26-192e300660df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c6ea11-sr6hx" Oct 30 13:04:24.491411 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:24.491384 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5639c55b-e419-4b24-9a26-192e300660df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19c6ea11-sr6hx\" (UID: \"5639c55b-e419-4b24-9a26-192e300660df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c6ea11-sr6hx" Oct 30 13:04:24.514034 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:24.514009 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t67n\" (UniqueName: \"kubernetes.io/projected/5639c55b-e419-4b24-9a26-192e300660df-kube-api-access-2t67n\") pod \"maas-api-revoke-sub-19c6ea11-sr6hx\" (UID: \"5639c55b-e419-4b24-9a26-192e300660df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c6ea11-sr6hx" Oct 30 13:04:24.580259 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:24.580161 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-183345e7-z2ghc" event={"ID":"784b2871-f3ab-4cdd-966d-be76d743fa2e","Type":"ContainerDied","Data":"dba57968ed51332c2baf5e19ca697ba7da2b97656b5e70b6d82a5b46bf4357dc"} Oct 30 13:04:24.580259 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:24.580190 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-183345e7-z2ghc" Oct 30 13:04:24.580259 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:24.580217 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dba57968ed51332c2baf5e19ca697ba7da2b97656b5e70b6d82a5b46bf4357dc" Oct 30 13:04:24.590234 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:24.590205 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-797rl\" (UniqueName: \"kubernetes.io/projected/f3463205-9bd3-431e-aa57-c6343263a1b8-kube-api-access-797rl\") pod \"maas-api-revoke-sub-27871d28-f4r7w\" (UID: \"f3463205-9bd3-431e-aa57-c6343263a1b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27871d28-f4r7w" Oct 30 13:04:24.590388 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:24.590311 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3463205-9bd3-431e-aa57-c6343263a1b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27871d28-f4r7w\" (UID: \"f3463205-9bd3-431e-aa57-c6343263a1b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27871d28-f4r7w" Oct 30 13:04:24.590581 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:24.590561 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3463205-9bd3-431e-aa57-c6343263a1b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27871d28-f4r7w\" (UID: \"f3463205-9bd3-431e-aa57-c6343263a1b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27871d28-f4r7w" Oct 30 13:04:24.591116 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:24.591097 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3463205-9bd3-431e-aa57-c6343263a1b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27871d28-f4r7w\" (UID: \"f3463205-9bd3-431e-aa57-c6343263a1b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27871d28-f4r7w" Oct 30 13:04:24.593005 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:24.592987 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3463205-9bd3-431e-aa57-c6343263a1b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27871d28-f4r7w\" (UID: \"f3463205-9bd3-431e-aa57-c6343263a1b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27871d28-f4r7w" Oct 30 13:04:24.613143 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:24.613106 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c6ea11-sr6hx" Oct 30 13:04:24.615555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:24.615521 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-797rl\" (UniqueName: \"kubernetes.io/projected/f3463205-9bd3-431e-aa57-c6343263a1b8-kube-api-access-797rl\") pod \"maas-api-revoke-sub-27871d28-f4r7w\" (UID: \"f3463205-9bd3-431e-aa57-c6343263a1b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27871d28-f4r7w" Oct 30 13:04:24.695747 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:24.695716 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27871d28-f4r7w" Oct 30 13:04:24.926607 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:04:24.926571 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5639c55b_e419_4b24_9a26_192e300660df.slice/crio-26278ed6f350b148e4f62b89e9b36a6d09d2979e37c067b911ad5dc4979c1aac WatchSource:0}: Error finding container 26278ed6f350b148e4f62b89e9b36a6d09d2979e37c067b911ad5dc4979c1aac: Status 404 returned error can't find the container with id 26278ed6f350b148e4f62b89e9b36a6d09d2979e37c067b911ad5dc4979c1aac Oct 30 13:04:24.927147 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:24.927119 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19c6ea11-sr6hx"] Oct 30 13:04:25.018466 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:04:25.018432 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3463205_9bd3_431e_aa57_c6343263a1b8.slice/crio-1d1dc21e4926f6a0a829693761f543952bd6b46ac5a12911ee759b484b69da0a WatchSource:0}: Error finding container 1d1dc21e4926f6a0a829693761f543952bd6b46ac5a12911ee759b484b69da0a: Status 404 returned error can't find the container with id 1d1dc21e4926f6a0a829693761f543952bd6b46ac5a12911ee759b484b69da0a Oct 30 13:04:25.018946 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:25.018924 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27871d28-f4r7w"] Oct 30 13:04:25.337400 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:25.337371 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:04:25.589064 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:25.588976 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c6ea11-sr6hx" event={"ID":"5639c55b-e419-4b24-9a26-192e300660df","Type":"ContainerStarted","Data":"bb77f2c13c6969a6965ecfac0052da2a23bb3170f06e2406ddec59ce3985762f"} Oct 30 13:04:25.589064 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:25.589042 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c6ea11-sr6hx" event={"ID":"5639c55b-e419-4b24-9a26-192e300660df","Type":"ContainerStarted","Data":"26278ed6f350b148e4f62b89e9b36a6d09d2979e37c067b911ad5dc4979c1aac"} Oct 30 13:04:25.590779 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:25.590749 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27871d28-f4r7w" event={"ID":"f3463205-9bd3-431e-aa57-c6343263a1b8","Type":"ContainerStarted","Data":"72d88217d611297879e67ee0c32880568bb27066f7ccb266a6447802910def8c"} Oct 30 13:04:25.590926 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:25.590796 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27871d28-f4r7w" event={"ID":"f3463205-9bd3-431e-aa57-c6343263a1b8","Type":"ContainerStarted","Data":"1d1dc21e4926f6a0a829693761f543952bd6b46ac5a12911ee759b484b69da0a"} Oct 30 13:04:25.645191 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:25.645124 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c6ea11-sr6hx" podStartSLOduration=1.64510237 podStartE2EDuration="1.64510237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:25.638599014 +0000 UTC m=+1164.242097474" watchObservedRunningTime="2026-10-30 13:04:25.64510237 +0000 UTC m=+1164.248600817" Oct 30 13:04:25.684927 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:25.684877 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27871d28-f4r7w" podStartSLOduration=1.684863963 podStartE2EDuration="1.684863963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:25.681889712 +0000 UTC m=+1164.285388159" watchObservedRunningTime="2026-10-30 13:04:25.684863963 +0000 UTC m=+1164.288362409" Oct 30 13:04:26.117296 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:04:26.116960 4931 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3463205_9bd3_431e_aa57_c6343263a1b8.slice/crio-72d88217d611297879e67ee0c32880568bb27066f7ccb266a6447802910def8c.scope\": RecentStats: unable to find data in memory cache]" Oct 30 13:04:26.635290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:26.631893 4931 generic.go:356] "Generic (PLEG): container finished" podID="5639c55b-e419-4b24-9a26-192e300660df" containerID="bb77f2c13c6969a6965ecfac0052da2a23bb3170f06e2406ddec59ce3985762f" exitCode=0 Oct 30 13:04:26.635290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:26.632127 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c6ea11-sr6hx" event={"ID":"5639c55b-e419-4b24-9a26-192e300660df","Type":"ContainerDied","Data":"bb77f2c13c6969a6965ecfac0052da2a23bb3170f06e2406ddec59ce3985762f"} Oct 30 13:04:26.635290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:26.634827 4931 generic.go:356] "Generic (PLEG): container finished" podID="f3463205-9bd3-431e-aa57-c6343263a1b8" containerID="72d88217d611297879e67ee0c32880568bb27066f7ccb266a6447802910def8c" exitCode=0 Oct 30 13:04:26.635290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:26.634914 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27871d28-f4r7w" event={"ID":"f3463205-9bd3-431e-aa57-c6343263a1b8","Type":"ContainerDied","Data":"72d88217d611297879e67ee0c32880568bb27066f7ccb266a6447802910def8c"} Oct 30 13:04:27.829393 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:27.829347 4931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" podUID="e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:27.907420 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:27.907372 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" podUID="5a504237-e5a3-491f-8f84-0419cf502f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:27.931395 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:27.931353 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" podUID="089cb620-913d-47c7-b8f7-9b2298c194bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 13:04:28.287342 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:28.267401 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" podUID="65799c14-8de4-4b4a-90df-7cd257e1e581" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:28.482298 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:28.470425 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c6ea11-sr6hx" Oct 30 13:04:28.661289 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:28.661119 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5639c55b-e419-4b24-9a26-192e300660df-kube-api-access-2t67n\" (UniqueName: \"kubernetes.io/projected/5639c55b-e419-4b24-9a26-192e300660df-kube-api-access-2t67n\") pod \"5639c55b-e419-4b24-9a26-192e300660df\" (UID: \"5639c55b-e419-4b24-9a26-192e300660df\") " Oct 30 13:04:28.661480 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:28.661308 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5639c55b-e419-4b24-9a26-192e300660df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5639c55b-e419-4b24-9a26-192e300660df-maas-api-xks-ca\") pod \"5639c55b-e419-4b24-9a26-192e300660df\" (UID: \"5639c55b-e419-4b24-9a26-192e300660df\") " Oct 30 13:04:28.661480 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:28.661432 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5639c55b-e419-4b24-9a26-192e300660df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5639c55b-e419-4b24-9a26-192e300660df-maas-api-openshift-ca\") pod \"5639c55b-e419-4b24-9a26-192e300660df\" (UID: \"5639c55b-e419-4b24-9a26-192e300660df\") " Oct 30 13:04:28.674618 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:28.669797 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5639c55b-e419-4b24-9a26-192e300660df-maas-api-openshift-ca" pod "5639c55b-e419-4b24-9a26-192e300660df" (UID: "5639c55b-e419-4b24-9a26-192e300660df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:28.674618 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:28.673316 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5639c55b-e419-4b24-9a26-192e300660df-kube-api-access-2t67n" pod "5639c55b-e419-4b24-9a26-192e300660df" (UID: "5639c55b-e419-4b24-9a26-192e300660df"). InnerVolumeSpecName "kube-api-access-2t67n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:28.674618 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:28.673316 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5639c55b-e419-4b24-9a26-192e300660df-maas-api-xks-ca" pod "5639c55b-e419-4b24-9a26-192e300660df" (UID: "5639c55b-e419-4b24-9a26-192e300660df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:28.676747 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:28.676724 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27871d28-f4r7w" Oct 30 13:04:28.682323 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:28.679800 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c6ea11-sr6hx" event={"ID":"5639c55b-e419-4b24-9a26-192e300660df","Type":"ContainerDied","Data":"26278ed6f350b148e4f62b89e9b36a6d09d2979e37c067b911ad5dc4979c1aac"} Oct 30 13:04:28.682323 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:28.679858 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26278ed6f350b148e4f62b89e9b36a6d09d2979e37c067b911ad5dc4979c1aac" Oct 30 13:04:28.682323 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:28.679928 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c6ea11-sr6hx" Oct 30 13:04:28.687508 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:28.687476 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27871d28-f4r7w" event={"ID":"f3463205-9bd3-431e-aa57-c6343263a1b8","Type":"ContainerDied","Data":"1d1dc21e4926f6a0a829693761f543952bd6b46ac5a12911ee759b484b69da0a"} Oct 30 13:04:28.687627 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:28.687514 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27871d28-f4r7w" Oct 30 13:04:28.687627 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:28.687530 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d1dc21e4926f6a0a829693761f543952bd6b46ac5a12911ee759b484b69da0a" Oct 30 13:04:28.764094 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:28.764057 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t67n\" (UniqueName: \"kubernetes.io/projected/5639c55b-e419-4b24-9a26-192e300660df-kube-api-access-2t67n\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:04:28.764378 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:28.764358 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5639c55b-e419-4b24-9a26-192e300660df-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:04:28.764513 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:28.764499 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5639c55b-e419-4b24-9a26-192e300660df-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:04:28.865758 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:28.865666 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3463205-9bd3-431e-aa57-c6343263a1b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3463205-9bd3-431e-aa57-c6343263a1b8-maas-api-openshift-ca\") pod \"f3463205-9bd3-431e-aa57-c6343263a1b8\" (UID: \"f3463205-9bd3-431e-aa57-c6343263a1b8\") " Oct 30 13:04:28.866183 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:28.865779 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3463205-9bd3-431e-aa57-c6343263a1b8-kube-api-access-797rl\" (UniqueName: \"kubernetes.io/projected/f3463205-9bd3-431e-aa57-c6343263a1b8-kube-api-access-797rl\") pod \"f3463205-9bd3-431e-aa57-c6343263a1b8\" (UID: \"f3463205-9bd3-431e-aa57-c6343263a1b8\") " Oct 30 13:04:28.866183 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:28.865985 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3463205-9bd3-431e-aa57-c6343263a1b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3463205-9bd3-431e-aa57-c6343263a1b8-maas-api-xks-ca\") pod \"f3463205-9bd3-431e-aa57-c6343263a1b8\" (UID: \"f3463205-9bd3-431e-aa57-c6343263a1b8\") " Oct 30 13:04:28.866183 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:28.866145 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3463205-9bd3-431e-aa57-c6343263a1b8-maas-api-openshift-ca" pod "f3463205-9bd3-431e-aa57-c6343263a1b8" (UID: "f3463205-9bd3-431e-aa57-c6343263a1b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:28.866644 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:28.866614 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3463205-9bd3-431e-aa57-c6343263a1b8-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:04:28.868332 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:28.868308 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3463205-9bd3-431e-aa57-c6343263a1b8-kube-api-access-797rl" pod "f3463205-9bd3-431e-aa57-c6343263a1b8" (UID: "f3463205-9bd3-431e-aa57-c6343263a1b8"). InnerVolumeSpecName "kube-api-access-797rl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:28.868834 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:28.868800 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3463205-9bd3-431e-aa57-c6343263a1b8-maas-api-xks-ca" pod "f3463205-9bd3-431e-aa57-c6343263a1b8" (UID: "f3463205-9bd3-431e-aa57-c6343263a1b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:28.968605 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:28.968562 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-797rl\" (UniqueName: \"kubernetes.io/projected/f3463205-9bd3-431e-aa57-c6343263a1b8-kube-api-access-797rl\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:04:28.968605 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:28.968606 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3463205-9bd3-431e-aa57-c6343263a1b8-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:04:30.741203 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:30.741168 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e05227f-59b7m"] Oct 30 13:04:30.741781 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:30.741765 4931 cpu_manager.go:414] "containerMap: removing container" podUID="5639c55b-e419-4b24-9a26-192e300660df" containerName="revoke-keys" Oct 30 13:04:30.741835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:30.741783 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5639c55b-e419-4b24-9a26-192e300660df" containerName="revoke-keys" Oct 30 13:04:30.741835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:30.741798 4931 cpu_manager.go:414] "containerMap: removing container" podUID="f3463205-9bd3-431e-aa57-c6343263a1b8" containerName="revoke-keys" Oct 30 13:04:30.741835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:30.741803 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3463205-9bd3-431e-aa57-c6343263a1b8" containerName="revoke-keys" Oct 30 13:04:30.741961 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:30.741905 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="5639c55b-e419-4b24-9a26-192e300660df" containerName="revoke-keys" Oct 30 13:04:30.741961 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:30.741924 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3463205-9bd3-431e-aa57-c6343263a1b8" containerName="revoke-keys" Oct 30 13:04:30.745726 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:30.745708 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e05227f-59b7m" Oct 30 13:04:30.775926 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:30.775898 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:04:30.775926 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:30.775915 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:04:30.780323 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:30.780299 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e05227f-59b7m"] Oct 30 13:04:30.789956 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:30.789925 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd9b9df5-4097-42db-94c8-1bbac6681a63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e05227f-59b7m\" (UID: \"dd9b9df5-4097-42db-94c8-1bbac6681a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e05227f-59b7m" Oct 30 13:04:30.790097 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:30.790075 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd9b9df5-4097-42db-94c8-1bbac6681a63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e05227f-59b7m\" (UID: \"dd9b9df5-4097-42db-94c8-1bbac6681a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e05227f-59b7m" Oct 30 13:04:30.790203 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:30.790185 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktxsv\" (UniqueName: \"kubernetes.io/projected/dd9b9df5-4097-42db-94c8-1bbac6681a63-kube-api-access-ktxsv\") pod \"maas-api-revoke-sub-4e05227f-59b7m\" (UID: \"dd9b9df5-4097-42db-94c8-1bbac6681a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e05227f-59b7m" Oct 30 13:04:30.891145 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:30.891110 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd9b9df5-4097-42db-94c8-1bbac6681a63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e05227f-59b7m\" (UID: \"dd9b9df5-4097-42db-94c8-1bbac6681a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e05227f-59b7m" Oct 30 13:04:30.891345 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:30.891197 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd9b9df5-4097-42db-94c8-1bbac6681a63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e05227f-59b7m\" (UID: \"dd9b9df5-4097-42db-94c8-1bbac6681a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e05227f-59b7m" Oct 30 13:04:30.891345 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:30.891232 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktxsv\" (UniqueName: \"kubernetes.io/projected/dd9b9df5-4097-42db-94c8-1bbac6681a63-kube-api-access-ktxsv\") pod \"maas-api-revoke-sub-4e05227f-59b7m\" (UID: \"dd9b9df5-4097-42db-94c8-1bbac6681a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e05227f-59b7m" Oct 30 13:04:30.891756 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:30.891730 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd9b9df5-4097-42db-94c8-1bbac6681a63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e05227f-59b7m\" (UID: \"dd9b9df5-4097-42db-94c8-1bbac6681a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e05227f-59b7m" Oct 30 13:04:30.893650 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:30.893626 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd9b9df5-4097-42db-94c8-1bbac6681a63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e05227f-59b7m\" (UID: \"dd9b9df5-4097-42db-94c8-1bbac6681a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e05227f-59b7m" Oct 30 13:04:30.916497 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:30.916470 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktxsv\" (UniqueName: \"kubernetes.io/projected/dd9b9df5-4097-42db-94c8-1bbac6681a63-kube-api-access-ktxsv\") pod \"maas-api-revoke-sub-4e05227f-59b7m\" (UID: \"dd9b9df5-4097-42db-94c8-1bbac6681a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e05227f-59b7m" Oct 30 13:04:31.059946 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:31.059868 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e05227f-59b7m" Oct 30 13:04:31.375455 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:31.375425 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e05227f-59b7m"] Oct 30 13:04:31.375571 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:04:31.375552 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd9b9df5_4097_42db_94c8_1bbac6681a63.slice/crio-0debacb05bbd4f75da8236803253a33d1abd5f84cf5eb27030965907304c384c WatchSource:0}: Error finding container 0debacb05bbd4f75da8236803253a33d1abd5f84cf5eb27030965907304c384c: Status 404 returned error can't find the container with id 0debacb05bbd4f75da8236803253a33d1abd5f84cf5eb27030965907304c384c Oct 30 13:04:31.710856 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:31.710823 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e05227f-59b7m" event={"ID":"dd9b9df5-4097-42db-94c8-1bbac6681a63","Type":"ContainerStarted","Data":"0debacb05bbd4f75da8236803253a33d1abd5f84cf5eb27030965907304c384c"} Oct 30 13:04:32.719603 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:32.719565 4931 generic.go:356] "Generic (PLEG): container finished" podID="dd9b9df5-4097-42db-94c8-1bbac6681a63" containerID="87bdb23de0a1d179688d66eb1f0a3c1c8af90b11744366b4e31d8bc9b8e0cb27" exitCode=0 Oct 30 13:04:32.720094 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:32.719646 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e05227f-59b7m" event={"ID":"dd9b9df5-4097-42db-94c8-1bbac6681a63","Type":"ContainerDied","Data":"87bdb23de0a1d179688d66eb1f0a3c1c8af90b11744366b4e31d8bc9b8e0cb27"} Oct 30 13:04:33.228489 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:33.228448 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-739cb84c-l9pfj"] Oct 30 13:04:33.232792 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:33.232773 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-739cb84c-l9pfj" Oct 30 13:04:33.255372 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:33.255337 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-739cb84c-l9pfj"] Oct 30 13:04:33.323140 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:33.323104 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2djtx\" (UniqueName: \"kubernetes.io/projected/4d7d4ac5-af2d-471b-a4c8-53daada47675-kube-api-access-2djtx\") pod \"maas-api-revoke-sub-739cb84c-l9pfj\" (UID: \"4d7d4ac5-af2d-471b-a4c8-53daada47675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-739cb84c-l9pfj" Oct 30 13:04:33.323400 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:33.323376 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d7d4ac5-af2d-471b-a4c8-53daada47675-maas-api-xks-ca\") pod \"maas-api-revoke-sub-739cb84c-l9pfj\" (UID: \"4d7d4ac5-af2d-471b-a4c8-53daada47675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-739cb84c-l9pfj" Oct 30 13:04:33.323502 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:33.323414 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d7d4ac5-af2d-471b-a4c8-53daada47675-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-739cb84c-l9pfj\" (UID: \"4d7d4ac5-af2d-471b-a4c8-53daada47675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-739cb84c-l9pfj" Oct 30 13:04:33.425118 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:33.425075 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d7d4ac5-af2d-471b-a4c8-53daada47675-maas-api-xks-ca\") pod \"maas-api-revoke-sub-739cb84c-l9pfj\" (UID: \"4d7d4ac5-af2d-471b-a4c8-53daada47675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-739cb84c-l9pfj" Oct 30 13:04:33.425325 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:33.425129 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d7d4ac5-af2d-471b-a4c8-53daada47675-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-739cb84c-l9pfj\" (UID: \"4d7d4ac5-af2d-471b-a4c8-53daada47675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-739cb84c-l9pfj" Oct 30 13:04:33.425325 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:33.425242 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2djtx\" (UniqueName: \"kubernetes.io/projected/4d7d4ac5-af2d-471b-a4c8-53daada47675-kube-api-access-2djtx\") pod \"maas-api-revoke-sub-739cb84c-l9pfj\" (UID: \"4d7d4ac5-af2d-471b-a4c8-53daada47675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-739cb84c-l9pfj" Oct 30 13:04:33.425783 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:33.425761 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d7d4ac5-af2d-471b-a4c8-53daada47675-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-739cb84c-l9pfj\" (UID: \"4d7d4ac5-af2d-471b-a4c8-53daada47675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-739cb84c-l9pfj" Oct 30 13:04:33.427597 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:33.427574 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d7d4ac5-af2d-471b-a4c8-53daada47675-maas-api-xks-ca\") pod \"maas-api-revoke-sub-739cb84c-l9pfj\" (UID: \"4d7d4ac5-af2d-471b-a4c8-53daada47675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-739cb84c-l9pfj" Oct 30 13:04:33.461034 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:33.460997 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2djtx\" (UniqueName: \"kubernetes.io/projected/4d7d4ac5-af2d-471b-a4c8-53daada47675-kube-api-access-2djtx\") pod \"maas-api-revoke-sub-739cb84c-l9pfj\" (UID: \"4d7d4ac5-af2d-471b-a4c8-53daada47675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-739cb84c-l9pfj" Oct 30 13:04:33.546731 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:33.546657 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-739cb84c-l9pfj" Oct 30 13:04:33.860880 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:33.860846 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-739cb84c-l9pfj"] Oct 30 13:04:33.866411 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:04:33.866371 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7d4ac5_af2d_471b_a4c8_53daada47675.slice/crio-afb43fdc5881f950f5e76b7680e4c2e7449d7ff064e2dd46ca3e0edda2776dcc WatchSource:0}: Error finding container afb43fdc5881f950f5e76b7680e4c2e7449d7ff064e2dd46ca3e0edda2776dcc: Status 404 returned error can't find the container with id afb43fdc5881f950f5e76b7680e4c2e7449d7ff064e2dd46ca3e0edda2776dcc Oct 30 13:04:33.879243 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:33.879216 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e05227f-59b7m" Oct 30 13:04:33.940333 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:33.935048 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd9b9df5-4097-42db-94c8-1bbac6681a63-kube-api-access-ktxsv\" (UniqueName: \"kubernetes.io/projected/dd9b9df5-4097-42db-94c8-1bbac6681a63-kube-api-access-ktxsv\") pod \"dd9b9df5-4097-42db-94c8-1bbac6681a63\" (UID: \"dd9b9df5-4097-42db-94c8-1bbac6681a63\") " Oct 30 13:04:33.940333 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:33.935522 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd9b9df5-4097-42db-94c8-1bbac6681a63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd9b9df5-4097-42db-94c8-1bbac6681a63-maas-api-xks-ca\") pod \"dd9b9df5-4097-42db-94c8-1bbac6681a63\" (UID: \"dd9b9df5-4097-42db-94c8-1bbac6681a63\") " Oct 30 13:04:33.940333 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:33.936140 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd9b9df5-4097-42db-94c8-1bbac6681a63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd9b9df5-4097-42db-94c8-1bbac6681a63-maas-api-openshift-ca\") pod \"dd9b9df5-4097-42db-94c8-1bbac6681a63\" (UID: \"dd9b9df5-4097-42db-94c8-1bbac6681a63\") " Oct 30 13:04:33.940333 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:33.936608 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd9b9df5-4097-42db-94c8-1bbac6681a63-maas-api-openshift-ca" pod "dd9b9df5-4097-42db-94c8-1bbac6681a63" (UID: "dd9b9df5-4097-42db-94c8-1bbac6681a63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:33.940333 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:33.937834 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd9b9df5-4097-42db-94c8-1bbac6681a63-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:04:34.033018 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:34.032977 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd9b9df5-4097-42db-94c8-1bbac6681a63-maas-api-xks-ca" pod "dd9b9df5-4097-42db-94c8-1bbac6681a63" (UID: "dd9b9df5-4097-42db-94c8-1bbac6681a63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:34.033018 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:34.033005 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd9b9df5-4097-42db-94c8-1bbac6681a63-kube-api-access-ktxsv" pod "dd9b9df5-4097-42db-94c8-1bbac6681a63" (UID: "dd9b9df5-4097-42db-94c8-1bbac6681a63"). InnerVolumeSpecName "kube-api-access-ktxsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:34.039066 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:34.039035 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd9b9df5-4097-42db-94c8-1bbac6681a63-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:04:34.039066 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:34.039065 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktxsv\" (UniqueName: \"kubernetes.io/projected/dd9b9df5-4097-42db-94c8-1bbac6681a63-kube-api-access-ktxsv\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:04:34.735530 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:34.735493 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e05227f-59b7m" event={"ID":"dd9b9df5-4097-42db-94c8-1bbac6681a63","Type":"ContainerDied","Data":"0debacb05bbd4f75da8236803253a33d1abd5f84cf5eb27030965907304c384c"} Oct 30 13:04:34.735530 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:34.735516 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e05227f-59b7m" Oct 30 13:04:34.735784 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:34.735559 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0debacb05bbd4f75da8236803253a33d1abd5f84cf5eb27030965907304c384c" Oct 30 13:04:34.737617 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:34.737583 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-739cb84c-l9pfj" event={"ID":"4d7d4ac5-af2d-471b-a4c8-53daada47675","Type":"ContainerStarted","Data":"a8aaf71ebc00a815442a4c3952434f814b30d77eb80accd3cfb10b6ad4ecf86b"} Oct 30 13:04:34.737749 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:34.737637 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-739cb84c-l9pfj" event={"ID":"4d7d4ac5-af2d-471b-a4c8-53daada47675","Type":"ContainerStarted","Data":"afb43fdc5881f950f5e76b7680e4c2e7449d7ff064e2dd46ca3e0edda2776dcc"} Oct 30 13:04:34.787136 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:34.787072 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-739cb84c-l9pfj" podStartSLOduration=1.787053379 podStartE2EDuration="1.787053379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:34.78232173 +0000 UTC m=+1173.385820173" watchObservedRunningTime="2026-10-30 13:04:34.787053379 +0000 UTC m=+1173.390551827" Oct 30 13:04:35.745744 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:35.745712 4931 generic.go:356] "Generic (PLEG): container finished" podID="4d7d4ac5-af2d-471b-a4c8-53daada47675" containerID="a8aaf71ebc00a815442a4c3952434f814b30d77eb80accd3cfb10b6ad4ecf86b" exitCode=0 Oct 30 13:04:35.746159 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:35.745792 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-739cb84c-l9pfj" event={"ID":"4d7d4ac5-af2d-471b-a4c8-53daada47675","Type":"ContainerDied","Data":"a8aaf71ebc00a815442a4c3952434f814b30d77eb80accd3cfb10b6ad4ecf86b"} Oct 30 13:04:36.906921 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:36.906890 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-739cb84c-l9pfj" Oct 30 13:04:36.975431 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:36.975393 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d7d4ac5-af2d-471b-a4c8-53daada47675-kube-api-access-2djtx\" (UniqueName: \"kubernetes.io/projected/4d7d4ac5-af2d-471b-a4c8-53daada47675-kube-api-access-2djtx\") pod \"4d7d4ac5-af2d-471b-a4c8-53daada47675\" (UID: \"4d7d4ac5-af2d-471b-a4c8-53daada47675\") " Oct 30 13:04:36.975573 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:36.975446 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d7d4ac5-af2d-471b-a4c8-53daada47675-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d7d4ac5-af2d-471b-a4c8-53daada47675-maas-api-openshift-ca\") pod \"4d7d4ac5-af2d-471b-a4c8-53daada47675\" (UID: \"4d7d4ac5-af2d-471b-a4c8-53daada47675\") " Oct 30 13:04:36.975632 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:36.975580 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d7d4ac5-af2d-471b-a4c8-53daada47675-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d7d4ac5-af2d-471b-a4c8-53daada47675-maas-api-xks-ca\") pod \"4d7d4ac5-af2d-471b-a4c8-53daada47675\" (UID: \"4d7d4ac5-af2d-471b-a4c8-53daada47675\") " Oct 30 13:04:36.975901 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:36.975865 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d7d4ac5-af2d-471b-a4c8-53daada47675-maas-api-openshift-ca" pod "4d7d4ac5-af2d-471b-a4c8-53daada47675" (UID: "4d7d4ac5-af2d-471b-a4c8-53daada47675"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:36.976226 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:36.976205 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d7d4ac5-af2d-471b-a4c8-53daada47675-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:04:36.978150 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:36.978123 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d7d4ac5-af2d-471b-a4c8-53daada47675-maas-api-xks-ca" pod "4d7d4ac5-af2d-471b-a4c8-53daada47675" (UID: "4d7d4ac5-af2d-471b-a4c8-53daada47675"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:36.983132 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:36.983110 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d7d4ac5-af2d-471b-a4c8-53daada47675-kube-api-access-2djtx" pod "4d7d4ac5-af2d-471b-a4c8-53daada47675" (UID: "4d7d4ac5-af2d-471b-a4c8-53daada47675"). InnerVolumeSpecName "kube-api-access-2djtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:37.078082 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:37.078009 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d7d4ac5-af2d-471b-a4c8-53daada47675-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:04:37.078082 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:37.078037 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2djtx\" (UniqueName: \"kubernetes.io/projected/4d7d4ac5-af2d-471b-a4c8-53daada47675-kube-api-access-2djtx\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:04:37.352552 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:04:37.352467 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:04:37.760654 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:37.760625 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-739cb84c-l9pfj" Oct 30 13:04:37.760804 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:37.760661 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-739cb84c-l9pfj" event={"ID":"4d7d4ac5-af2d-471b-a4c8-53daada47675","Type":"ContainerDied","Data":"afb43fdc5881f950f5e76b7680e4c2e7449d7ff064e2dd46ca3e0edda2776dcc"} Oct 30 13:04:37.760804 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:37.760716 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afb43fdc5881f950f5e76b7680e4c2e7449d7ff064e2dd46ca3e0edda2776dcc" Oct 30 13:04:43.404352 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:43.404316 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz"] Oct 30 13:04:43.406249 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:43.406218 4931 cpu_manager.go:414] "containerMap: removing container" podUID="4d7d4ac5-af2d-471b-a4c8-53daada47675" containerName="revoke-keys" Oct 30 13:04:43.406439 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:43.406424 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d7d4ac5-af2d-471b-a4c8-53daada47675" containerName="revoke-keys" Oct 30 13:04:43.406547 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:43.406537 4931 cpu_manager.go:414] "containerMap: removing container" podUID="dd9b9df5-4097-42db-94c8-1bbac6681a63" containerName="revoke-keys" Oct 30 13:04:43.406616 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:43.406607 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd9b9df5-4097-42db-94c8-1bbac6681a63" containerName="revoke-keys" Oct 30 13:04:43.407038 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:43.407019 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd9b9df5-4097-42db-94c8-1bbac6681a63" containerName="revoke-keys" Oct 30 13:04:43.407186 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:43.407175 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d7d4ac5-af2d-471b-a4c8-53daada47675" containerName="revoke-keys" Oct 30 13:04:43.411834 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:43.411808 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" Oct 30 13:04:43.434091 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:43.434057 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz"] Oct 30 13:04:43.443238 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:43.443216 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-7a3603\"/\"distinct-w5-7a3603-kserve-self-signed-certs\"" Oct 30 13:04:43.572969 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:43.572924 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3b342db-9a01-46ad-b476-bfc912d9ce1c-model-cache\") pod \"distinct-w5-7a3603-kserve-f7997fc8d-n9bfz\" (UID: \"f3b342db-9a01-46ad-b476-bfc912d9ce1c\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" Oct 30 13:04:43.573149 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:43.572979 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3b342db-9a01-46ad-b476-bfc912d9ce1c-tls-certs\") pod \"distinct-w5-7a3603-kserve-f7997fc8d-n9bfz\" (UID: \"f3b342db-9a01-46ad-b476-bfc912d9ce1c\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" Oct 30 13:04:43.573206 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:43.573180 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3b342db-9a01-46ad-b476-bfc912d9ce1c-tmp-dir\") pod \"distinct-w5-7a3603-kserve-f7997fc8d-n9bfz\" (UID: \"f3b342db-9a01-46ad-b476-bfc912d9ce1c\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" Oct 30 13:04:43.573260 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:43.573229 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3b342db-9a01-46ad-b476-bfc912d9ce1c-home\") pod \"distinct-w5-7a3603-kserve-f7997fc8d-n9bfz\" (UID: \"f3b342db-9a01-46ad-b476-bfc912d9ce1c\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" Oct 30 13:04:43.573438 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:43.573420 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw5v9\" (UniqueName: \"kubernetes.io/projected/f3b342db-9a01-46ad-b476-bfc912d9ce1c-kube-api-access-pw5v9\") pod \"distinct-w5-7a3603-kserve-f7997fc8d-n9bfz\" (UID: \"f3b342db-9a01-46ad-b476-bfc912d9ce1c\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" Oct 30 13:04:43.573489 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:43.573474 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3b342db-9a01-46ad-b476-bfc912d9ce1c-dshm\") pod \"distinct-w5-7a3603-kserve-f7997fc8d-n9bfz\" (UID: \"f3b342db-9a01-46ad-b476-bfc912d9ce1c\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" Oct 30 13:04:43.675007 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:43.674923 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3b342db-9a01-46ad-b476-bfc912d9ce1c-dshm\") pod \"distinct-w5-7a3603-kserve-f7997fc8d-n9bfz\" (UID: \"f3b342db-9a01-46ad-b476-bfc912d9ce1c\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" Oct 30 13:04:43.675176 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:43.675015 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3b342db-9a01-46ad-b476-bfc912d9ce1c-model-cache\") pod \"distinct-w5-7a3603-kserve-f7997fc8d-n9bfz\" (UID: \"f3b342db-9a01-46ad-b476-bfc912d9ce1c\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" Oct 30 13:04:43.675176 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:43.675046 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3b342db-9a01-46ad-b476-bfc912d9ce1c-tls-certs\") pod \"distinct-w5-7a3603-kserve-f7997fc8d-n9bfz\" (UID: \"f3b342db-9a01-46ad-b476-bfc912d9ce1c\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" Oct 30 13:04:43.675328 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:43.675232 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3b342db-9a01-46ad-b476-bfc912d9ce1c-tmp-dir\") pod \"distinct-w5-7a3603-kserve-f7997fc8d-n9bfz\" (UID: \"f3b342db-9a01-46ad-b476-bfc912d9ce1c\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" Oct 30 13:04:43.675328 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:43.675301 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3b342db-9a01-46ad-b476-bfc912d9ce1c-home\") pod \"distinct-w5-7a3603-kserve-f7997fc8d-n9bfz\" (UID: \"f3b342db-9a01-46ad-b476-bfc912d9ce1c\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" Oct 30 13:04:43.675471 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:43.675451 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw5v9\" (UniqueName: \"kubernetes.io/projected/f3b342db-9a01-46ad-b476-bfc912d9ce1c-kube-api-access-pw5v9\") pod \"distinct-w5-7a3603-kserve-f7997fc8d-n9bfz\" (UID: \"f3b342db-9a01-46ad-b476-bfc912d9ce1c\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" Oct 30 13:04:43.675929 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:43.675887 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3b342db-9a01-46ad-b476-bfc912d9ce1c-tmp-dir\") pod \"distinct-w5-7a3603-kserve-f7997fc8d-n9bfz\" (UID: \"f3b342db-9a01-46ad-b476-bfc912d9ce1c\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" Oct 30 13:04:43.676130 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:43.676107 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3b342db-9a01-46ad-b476-bfc912d9ce1c-home\") pod \"distinct-w5-7a3603-kserve-f7997fc8d-n9bfz\" (UID: \"f3b342db-9a01-46ad-b476-bfc912d9ce1c\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" Oct 30 13:04:43.676259 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:43.676234 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3b342db-9a01-46ad-b476-bfc912d9ce1c-model-cache\") pod \"distinct-w5-7a3603-kserve-f7997fc8d-n9bfz\" (UID: \"f3b342db-9a01-46ad-b476-bfc912d9ce1c\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" Oct 30 13:04:43.678231 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:43.678209 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3b342db-9a01-46ad-b476-bfc912d9ce1c-dshm\") pod \"distinct-w5-7a3603-kserve-f7997fc8d-n9bfz\" (UID: \"f3b342db-9a01-46ad-b476-bfc912d9ce1c\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" Oct 30 13:04:43.678595 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:43.678397 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3b342db-9a01-46ad-b476-bfc912d9ce1c-tls-certs\") pod \"distinct-w5-7a3603-kserve-f7997fc8d-n9bfz\" (UID: \"f3b342db-9a01-46ad-b476-bfc912d9ce1c\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" Oct 30 13:04:43.721326 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:43.713461 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw5v9\" (UniqueName: \"kubernetes.io/projected/f3b342db-9a01-46ad-b476-bfc912d9ce1c-kube-api-access-pw5v9\") pod \"distinct-w5-7a3603-kserve-f7997fc8d-n9bfz\" (UID: \"f3b342db-9a01-46ad-b476-bfc912d9ce1c\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" Oct 30 13:04:43.725318 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:43.725288 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" Oct 30 13:04:44.067730 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:04:44.067700 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3b342db_9a01_46ad_b476_bfc912d9ce1c.slice/crio-96c28132141ca5d05b96760ebec98321de64feca067761e14ac013b3c1c8c664 WatchSource:0}: Error finding container 96c28132141ca5d05b96760ebec98321de64feca067761e14ac013b3c1c8c664: Status 404 returned error can't find the container with id 96c28132141ca5d05b96760ebec98321de64feca067761e14ac013b3c1c8c664 Oct 30 13:04:44.067890 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:44.067869 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz"] Oct 30 13:04:44.822984 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:44.822940 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" event={"ID":"f3b342db-9a01-46ad-b476-bfc912d9ce1c","Type":"ContainerStarted","Data":"79e26cbf888e58bce0368a612e9795d8d1056f87ab4e441c8a042ac3b5ac7767"} Oct 30 13:04:44.823427 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:44.822998 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" event={"ID":"f3b342db-9a01-46ad-b476-bfc912d9ce1c","Type":"ContainerStarted","Data":"96c28132141ca5d05b96760ebec98321de64feca067761e14ac013b3c1c8c664"} Oct 30 13:04:44.918286 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:44.906467 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" podStartSLOduration=1.906447875 podStartE2EDuration="1.906447875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:44.896944542 +0000 UTC m=+1183.500443021" watchObservedRunningTime="2026-10-30 13:04:44.906447875 +0000 UTC m=+1183.509946342" Oct 30 13:04:45.728374 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:45.728341 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" Oct 30 13:04:47.147419 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:47.147366 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" podUID="65799c14-8de4-4b4a-90df-7cd257e1e581" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:47.347626 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:47.347574 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" podUID="7ad1989a-c73f-4647-9e50-855605bdb08e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:47.464445 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:47.464397 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" podUID="d345fdd5-407b-4162-a2cd-64a40068cb3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:47.477209 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:47.477163 4931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" podUID="d345fdd5-407b-4162-a2cd-64a40068cb3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:47.497375 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:47.497337 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" podUID="606b252b-881c-4d40-8ba9-fc4ed937a995" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 13:04:47.509386 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:47.509349 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" podUID="e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:47.527418 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:47.527378 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" podUID="64468005-61ea-4f2a-b72c-9c30c6a4d686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:47.712424 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:47.712359 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" podUID="0c5a9f00-6d5b-4860-a210-2fec24afb322" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:47.828004 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:47.827903 4931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" podUID="e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:47.964836 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:47.964787 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" podUID="5a504237-e5a3-491f-8f84-0419cf502f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 13:04:47.967103 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:47.965096 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" podUID="089cb620-913d-47c7-b8f7-9b2298c194bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:47.982658 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:47.966693 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" podUID="b1e86eea-4824-4c50-8ad3-2a15c3365c3d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 13:04:48.148545 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:48.148412 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" podUID="65799c14-8de4-4b4a-90df-7cd257e1e581" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 13:04:48.149968 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:48.149948 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:04:48.434345 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:48.425674 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" podUID="7ad1989a-c73f-4647-9e50-855605bdb08e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 13:04:48.461299 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:48.434840 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:04:48.495583 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:48.495536 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" podUID="d345fdd5-407b-4162-a2cd-64a40068cb3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:48.496764 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:48.496738 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 13:04:48.535290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:48.524393 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" podUID="e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:48.535290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:48.524804 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" podUID="9577f571-30cd-4043-a3e4-72bf95837552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 13:04:48.535290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:48.525133 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:04:48.636495 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:48.636439 4931 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:04:48.636840 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:48.636809 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" podUID="fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:48.672476 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:48.672385 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" podUID="606b252b-881c-4d40-8ba9-fc4ed937a995" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:48.722293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:48.719460 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:04:48.722293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:48.719531 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 13:04:48.742290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:48.732511 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" podUID="0c5a9f00-6d5b-4860-a210-2fec24afb322" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:48.742290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:48.732937 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 13:04:48.971291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:48.971032 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" podUID="5a504237-e5a3-491f-8f84-0419cf502f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 13:04:48.996339 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:48.971803 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:04:49.031296 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:49.030756 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" podUID="089cb620-913d-47c7-b8f7-9b2298c194bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:49.031296 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:49.031144 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 13:04:49.054299 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:49.037288 4931 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:04:49.054299 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:49.037337 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" podUID="655adb9b-cbfb-476b-9c20-600e99055bd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:49.352344 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:49.340120 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 13:04:49.426003 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:49.425964 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" podUID="7ad1989a-c73f-4647-9e50-855605bdb08e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 13:04:49.531438 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:49.531381 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" podUID="e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:49.596291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:49.593905 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" podUID="64468005-61ea-4f2a-b72c-9c30c6a4d686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:49.603875 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:49.603789 4931 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:04:49.604099 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:49.604069 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" podUID="fc65620b-073b-4e13-82da-38526fb90da3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:49.653038 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:49.652995 4931 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:04:49.653351 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:49.653314 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" podUID="16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:49.712486 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:49.712439 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" podUID="606b252b-881c-4d40-8ba9-fc4ed937a995" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:49.741929 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:49.713242 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" podUID="65799c14-8de4-4b4a-90df-7cd257e1e581" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:49.743928 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:49.742360 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" podUID="0c5a9f00-6d5b-4860-a210-2fec24afb322" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:49.796293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:49.772209 4931 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:04:49.796293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:49.772290 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" podUID="2e8d41cb-daa2-4be3-b42d-90a1951a1fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:49.796293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:49.786780 4931 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:04:49.796293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:49.786851 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" podUID="3680a3d2-0b13-40df-aeeb-6a924495f8d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:49.934466 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:49.934360 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" podUID="b1e86eea-4824-4c50-8ad3-2a15c3365c3d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:50.090126 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:49.983151 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" podUID="5a504237-e5a3-491f-8f84-0419cf502f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:50.090647 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:50.044435 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" podUID="089cb620-913d-47c7-b8f7-9b2298c194bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:50.254288 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:50.252708 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 13:04:50.415400 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:50.360121 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" podUID="d345fdd5-407b-4162-a2cd-64a40068cb3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:50.481310 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:50.448360 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" podUID="7ad1989a-c73f-4647-9e50-855605bdb08e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:50.503617 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:50.501174 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" podUID="9577f571-30cd-4043-a3e4-72bf95837552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:50.539346 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:50.534571 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" podUID="e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:50.607471 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:50.597592 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" podUID="64468005-61ea-4f2a-b72c-9c30c6a4d686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:50.607471 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:50.597987 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 13:04:50.607471 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:50.604582 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:04:50.607471 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:50.606032 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:04:50.607471 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:50.606241 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 13:04:50.625592 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:50.625570 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:04:50.629466 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:50.629352 4931 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:04:50.629466 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:50.629400 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" podUID="4e1ed676-cd86-4b17-97af-1a5a501f6274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:50.685680 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:50.685640 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 13:04:50.812343 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:50.808831 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 13:04:53.726554 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:53.726522 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" Oct 30 13:04:53.738836 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:53.738811 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" Oct 30 13:04:54.157352 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:54.157253 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" Oct 30 13:04:55.486673 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:55.486636 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a5ac23b-8jlkj"] Oct 30 13:04:55.491982 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:55.491953 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5ac23b-8jlkj" Oct 30 13:04:55.530010 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:55.529960 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:04:55.542138 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:55.542107 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:04:55.549340 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:55.549304 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a5ac23b-8jlkj"] Oct 30 13:04:55.623563 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:55.623522 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28xm6\" (UniqueName: \"kubernetes.io/projected/bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2-kube-api-access-28xm6\") pod \"maas-api-revoke-sub-6a5ac23b-8jlkj\" (UID: \"bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5ac23b-8jlkj" Oct 30 13:04:55.623761 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:55.623580 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a5ac23b-8jlkj\" (UID: \"bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5ac23b-8jlkj" Oct 30 13:04:55.623761 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:55.623734 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a5ac23b-8jlkj\" (UID: \"bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5ac23b-8jlkj" Oct 30 13:04:55.725179 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:55.725139 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28xm6\" (UniqueName: \"kubernetes.io/projected/bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2-kube-api-access-28xm6\") pod \"maas-api-revoke-sub-6a5ac23b-8jlkj\" (UID: \"bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5ac23b-8jlkj" Oct 30 13:04:55.725179 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:55.725181 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a5ac23b-8jlkj\" (UID: \"bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5ac23b-8jlkj" Oct 30 13:04:55.725538 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:55.725214 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a5ac23b-8jlkj\" (UID: \"bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5ac23b-8jlkj" Oct 30 13:04:55.725897 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:55.725873 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a5ac23b-8jlkj\" (UID: \"bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5ac23b-8jlkj" Oct 30 13:04:55.727671 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:55.727646 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a5ac23b-8jlkj\" (UID: \"bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5ac23b-8jlkj" Oct 30 13:04:55.757897 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:55.757842 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28xm6\" (UniqueName: \"kubernetes.io/projected/bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2-kube-api-access-28xm6\") pod \"maas-api-revoke-sub-6a5ac23b-8jlkj\" (UID: \"bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5ac23b-8jlkj" Oct 30 13:04:55.807124 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:55.807077 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5ac23b-8jlkj" Oct 30 13:04:56.155636 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:04:56.155595 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd76ca70_10f3_4cdb_b1d0_f8b1a34430e2.slice/crio-f36cb60a9531674d6f0d77a9506dbf77c9ace6bb55f6274d0d2b79ce18b64696 WatchSource:0}: Error finding container f36cb60a9531674d6f0d77a9506dbf77c9ace6bb55f6274d0d2b79ce18b64696: Status 404 returned error can't find the container with id f36cb60a9531674d6f0d77a9506dbf77c9ace6bb55f6274d0d2b79ce18b64696 Oct 30 13:04:56.155871 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:56.155846 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a5ac23b-8jlkj"] Oct 30 13:04:57.171151 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:57.171121 4931 generic.go:356] "Generic (PLEG): container finished" podID="bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2" containerID="60ee994d1b964ffda9de6f562cf2c97bba845388a1188f66c973fd478d4319f8" exitCode=0 Oct 30 13:04:57.171568 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:57.171209 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5ac23b-8jlkj" event={"ID":"bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2","Type":"ContainerDied","Data":"60ee994d1b964ffda9de6f562cf2c97bba845388a1188f66c973fd478d4319f8"} Oct 30 13:04:57.171568 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:57.171294 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5ac23b-8jlkj" event={"ID":"bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2","Type":"ContainerStarted","Data":"f36cb60a9531674d6f0d77a9506dbf77c9ace6bb55f6274d0d2b79ce18b64696"} Oct 30 13:04:58.342558 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:58.342531 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5ac23b-8jlkj" Oct 30 13:04:58.463123 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:58.463092 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2-maas-api-xks-ca\") pod \"bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2\" (UID: \"bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2\") " Oct 30 13:04:58.463342 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:58.463226 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2-kube-api-access-28xm6\" (UniqueName: \"kubernetes.io/projected/bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2-kube-api-access-28xm6\") pod \"bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2\" (UID: \"bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2\") " Oct 30 13:04:58.463342 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:58.463328 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2-maas-api-openshift-ca\") pod \"bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2\" (UID: \"bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2\") " Oct 30 13:04:58.463702 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:58.463672 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2-maas-api-openshift-ca" pod "bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2" (UID: "bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:58.463827 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:58.463811 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.465302 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:58.465253 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2-maas-api-xks-ca" pod "bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2" (UID: "bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:58.465409 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:58.465386 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2-kube-api-access-28xm6" pod "bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2" (UID: "bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2"). InnerVolumeSpecName "kube-api-access-28xm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:58.565613 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:58.565579 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.565613 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:58.565608 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28xm6\" (UniqueName: \"kubernetes.io/projected/bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2-kube-api-access-28xm6\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:04:59.187479 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:59.187446 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5ac23b-8jlkj" event={"ID":"bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2","Type":"ContainerDied","Data":"f36cb60a9531674d6f0d77a9506dbf77c9ace6bb55f6274d0d2b79ce18b64696"} Oct 30 13:04:59.187652 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:59.187479 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5ac23b-8jlkj" Oct 30 13:04:59.187652 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:04:59.187496 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f36cb60a9531674d6f0d77a9506dbf77c9ace6bb55f6274d0d2b79ce18b64696" Oct 30 13:05:01.667846 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:01.667818 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/2.log" Oct 30 13:05:01.673345 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:01.673322 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/2.log" Oct 30 13:05:03.156936 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:03.156897 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:05:05.749290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:05.748673 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" podUID="0c5a9f00-6d5b-4860-a210-2fec24afb322" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:05.848294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:05.817410 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" podUID="64468005-61ea-4f2a-b72c-9c30c6a4d686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:05.875156 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:05.874495 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" podUID="b1e86eea-4824-4c50-8ad3-2a15c3365c3d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 13:05:06.163064 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:06.162957 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" podUID="f3b342db-9a01-46ad-b476-bfc912d9ce1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 30 13:05:06.274986 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:06.274939 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" podUID="606b252b-881c-4d40-8ba9-fc4ed937a995" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 13:05:06.342427 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:06.342374 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" podUID="d345fdd5-407b-4162-a2cd-64a40068cb3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:06.505409 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:06.505358 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" podUID="9577f571-30cd-4043-a3e4-72bf95837552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:06.535395 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:06.535343 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" podUID="e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 13:05:06.576290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:06.575401 4931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" podUID="5a504237-e5a3-491f-8f84-0419cf502f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:06.617480 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:06.617415 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" podUID="7ad1989a-c73f-4647-9e50-855605bdb08e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:06.623609 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:06.623507 4931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" podUID="b1e86eea-4824-4c50-8ad3-2a15c3365c3d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:06.713292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:06.705365 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" podUID="089cb620-913d-47c7-b8f7-9b2298c194bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:06.728308 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:06.713589 4931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" podUID="089cb620-913d-47c7-b8f7-9b2298c194bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:06.751403 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:06.738400 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" podUID="65799c14-8de4-4b4a-90df-7cd257e1e581" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 13:05:06.782357 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:06.760767 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" podUID="0c5a9f00-6d5b-4860-a210-2fec24afb322" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:06.782357 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:06.761171 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 13:05:06.840412 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:06.840366 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" podUID="64468005-61ea-4f2a-b72c-9c30c6a4d686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 13:05:06.841330 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:06.841305 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 13:05:06.892922 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:06.874710 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" podUID="b1e86eea-4824-4c50-8ad3-2a15c3365c3d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:06.892922 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:06.875145 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" Oct 30 13:05:07.375450 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:07.375396 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" podUID="606b252b-881c-4d40-8ba9-fc4ed937a995" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 13:05:07.635477 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:07.635371 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" podUID="7ad1989a-c73f-4647-9e50-855605bdb08e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 13:05:07.636994 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:07.375680 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" podUID="d345fdd5-407b-4162-a2cd-64a40068cb3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 13:05:07.638337 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:07.638315 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 13:05:07.638555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:07.638538 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:05:07.638743 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:07.638728 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 13:05:07.665005 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:07.501438 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" podUID="5a504237-e5a3-491f-8f84-0419cf502f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 13:05:07.685173 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:07.462560 4931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" podUID="0c5a9f00-6d5b-4860-a210-2fec24afb322" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": context deadline exceeded" Oct 30 13:05:07.686179 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:07.551597 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" podUID="9577f571-30cd-4043-a3e4-72bf95837552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 13:05:07.686652 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:07.551706 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" podUID="e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 13:05:07.686809 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:07.599647 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" podUID="f3b342db-9a01-46ad-b476-bfc912d9ce1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:07.708107 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:07.708073 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:05:07.749306 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:07.734250 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" podUID="089cb620-913d-47c7-b8f7-9b2298c194bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:08.048291 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:05:08.045223 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:05:08.050824 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:08.050782 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" podUID="0c5a9f00-6d5b-4860-a210-2fec24afb322" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:08.219332 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:08.051179 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" podUID="65799c14-8de4-4b4a-90df-7cd257e1e581" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:08.219332 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:08.105456 4931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" podUID="e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:08.219332 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:08.128015 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" podUID="64468005-61ea-4f2a-b72c-9c30c6a4d686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 13:05:08.219332 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:08.180431 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" podUID="b1e86eea-4824-4c50-8ad3-2a15c3365c3d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:08.328348 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:08.300817 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:05:08.328348 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:08.300878 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" Oct 30 13:05:08.328348 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:08.300912 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 13:05:08.328348 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:08.300960 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:05:08.447742 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:08.447712 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 13:05:08.591357 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:08.590820 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 13:05:08.637500 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:08.637422 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" podUID="7ad1989a-c73f-4647-9e50-855605bdb08e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 13:05:08.704680 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:08.702990 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" podUID="9577f571-30cd-4043-a3e4-72bf95837552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:08.704680 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:08.703413 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" podUID="5a504237-e5a3-491f-8f84-0419cf502f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 13:05:08.704680 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:08.703771 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:05:08.733326 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:08.710797 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" podUID="f3b342db-9a01-46ad-b476-bfc912d9ce1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:08.761306 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:08.735091 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" podUID="e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:08.761306 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:08.749349 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" podUID="089cb620-913d-47c7-b8f7-9b2298c194bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:08.835416 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:08.831123 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 13:05:08.835416 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:08.831188 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 13:05:08.886391 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:08.885890 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:05:09.084291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:09.067393 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:05:09.217542 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:09.215002 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" podUID="b1e86eea-4824-4c50-8ad3-2a15c3365c3d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:09.218012 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:09.217956 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:05:09.219010 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:09.218956 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" Oct 30 13:05:09.425352 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:09.425322 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" Oct 30 13:05:09.520789 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:09.455782 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:05:09.931706 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:09.931600 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:05:09.931863 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:09.931820 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 13:05:25.139719 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.139682 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv"] Oct 30 13:05:25.141129 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.141098 4931 cpu_manager.go:414] "containerMap: removing container" podUID="bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2" containerName="revoke-keys" Oct 30 13:05:25.141129 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.141128 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2" containerName="revoke-keys" Oct 30 13:05:25.141397 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.141339 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2" containerName="revoke-keys" Oct 30 13:05:25.145358 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.145338 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" Oct 30 13:05:25.189620 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.189582 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv"] Oct 30 13:05:25.309868 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.309827 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78d440f8-6cfb-4f8e-905e-228d07bdcec9-istio-podinfo\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" Oct 30 13:05:25.310064 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.310013 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d440f8-6cfb-4f8e-905e-228d07bdcec9-credential-socket\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" Oct 30 13:05:25.310064 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.310049 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78d440f8-6cfb-4f8e-905e-228d07bdcec9-istio-token\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" Oct 30 13:05:25.310151 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.310077 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rz44\" (UniqueName: \"kubernetes.io/projected/78d440f8-6cfb-4f8e-905e-228d07bdcec9-kube-api-access-6rz44\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" Oct 30 13:05:25.310244 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.310221 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d440f8-6cfb-4f8e-905e-228d07bdcec9-workload-socket\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" Oct 30 13:05:25.310355 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.310256 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78d440f8-6cfb-4f8e-905e-228d07bdcec9-istio-data\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" Oct 30 13:05:25.310424 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.310368 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78d440f8-6cfb-4f8e-905e-228d07bdcec9-istiod-ca-cert\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" Oct 30 13:05:25.310478 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.310421 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78d440f8-6cfb-4f8e-905e-228d07bdcec9-workload-certs\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" Oct 30 13:05:25.310478 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.310449 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78d440f8-6cfb-4f8e-905e-228d07bdcec9-istio-envoy\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" Oct 30 13:05:25.411602 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.411519 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d440f8-6cfb-4f8e-905e-228d07bdcec9-credential-socket\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" Oct 30 13:05:25.411602 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.411560 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78d440f8-6cfb-4f8e-905e-228d07bdcec9-istio-token\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" Oct 30 13:05:25.411828 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.411599 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rz44\" (UniqueName: \"kubernetes.io/projected/78d440f8-6cfb-4f8e-905e-228d07bdcec9-kube-api-access-6rz44\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" Oct 30 13:05:25.411828 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.411693 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d440f8-6cfb-4f8e-905e-228d07bdcec9-workload-socket\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" Oct 30 13:05:25.411930 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.411834 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78d440f8-6cfb-4f8e-905e-228d07bdcec9-istio-data\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" Oct 30 13:05:25.411930 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.411894 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78d440f8-6cfb-4f8e-905e-228d07bdcec9-istiod-ca-cert\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" Oct 30 13:05:25.412065 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.411925 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d440f8-6cfb-4f8e-905e-228d07bdcec9-credential-socket\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" Oct 30 13:05:25.412065 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.411937 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78d440f8-6cfb-4f8e-905e-228d07bdcec9-workload-certs\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" Oct 30 13:05:25.412065 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.412004 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d440f8-6cfb-4f8e-905e-228d07bdcec9-workload-socket\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" Oct 30 13:05:25.412205 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.412185 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78d440f8-6cfb-4f8e-905e-228d07bdcec9-istio-data\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" Oct 30 13:05:25.412245 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.412217 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78d440f8-6cfb-4f8e-905e-228d07bdcec9-workload-certs\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" Oct 30 13:05:25.412352 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.412316 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78d440f8-6cfb-4f8e-905e-228d07bdcec9-istio-envoy\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" Oct 30 13:05:25.412446 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.412429 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78d440f8-6cfb-4f8e-905e-228d07bdcec9-istio-podinfo\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" Oct 30 13:05:25.412640 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.412621 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78d440f8-6cfb-4f8e-905e-228d07bdcec9-istiod-ca-cert\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" Oct 30 13:05:25.414515 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.414486 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78d440f8-6cfb-4f8e-905e-228d07bdcec9-istio-envoy\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" Oct 30 13:05:25.414718 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.414700 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78d440f8-6cfb-4f8e-905e-228d07bdcec9-istio-podinfo\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" Oct 30 13:05:25.443361 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.443335 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78d440f8-6cfb-4f8e-905e-228d07bdcec9-istio-token\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" Oct 30 13:05:25.443675 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.443654 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rz44\" (UniqueName: \"kubernetes.io/projected/78d440f8-6cfb-4f8e-905e-228d07bdcec9-kube-api-access-6rz44\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" Oct 30 13:05:25.460773 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.460750 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j"] Oct 30 13:05:25.461135 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.461115 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" Oct 30 13:05:25.465593 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.465578 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" Oct 30 13:05:25.499147 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.499101 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j"] Oct 30 13:05:25.523638 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.523609 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz"] Oct 30 13:05:25.527882 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.527854 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" Oct 30 13:05:25.580882 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.580848 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz"] Oct 30 13:05:25.615300 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.615249 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-credential-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" Oct 30 13:05:25.615609 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.615477 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-istio-token\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" Oct 30 13:05:25.615724 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.615629 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-istio-envoy\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" Oct 30 13:05:25.615724 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.615655 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-istio-data\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" Oct 30 13:05:25.615724 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.615681 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-workload-certs\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" Oct 30 13:05:25.615900 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.615789 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-workload-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" Oct 30 13:05:25.615900 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.615858 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-istiod-ca-cert\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" Oct 30 13:05:25.616007 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.615893 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fppzm\" (UniqueName: \"kubernetes.io/projected/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-kube-api-access-fppzm\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" Oct 30 13:05:25.616007 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.615946 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-istio-podinfo\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" Oct 30 13:05:25.717826 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.717793 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fppzm\" (UniqueName: \"kubernetes.io/projected/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-kube-api-access-fppzm\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" Oct 30 13:05:25.717826 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.717839 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-istio-podinfo\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" Oct 30 13:05:25.718117 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.717884 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b656366-2609-4928-83f1-18bc76cb1c48-istio-podinfo\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" Oct 30 13:05:25.718117 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.717905 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b656366-2609-4928-83f1-18bc76cb1c48-workload-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" Oct 30 13:05:25.718117 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.718095 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b656366-2609-4928-83f1-18bc76cb1c48-workload-certs\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" Oct 30 13:05:25.718334 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.718165 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b656366-2609-4928-83f1-18bc76cb1c48-istio-token\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" Oct 30 13:05:25.718334 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.718215 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-istio-envoy\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" Oct 30 13:05:25.718334 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.718260 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-workload-certs\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" Oct 30 13:05:25.718334 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.718327 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b656366-2609-4928-83f1-18bc76cb1c48-credential-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" Oct 30 13:05:25.718538 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.718374 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2nfh\" (UniqueName: \"kubernetes.io/projected/0b656366-2609-4928-83f1-18bc76cb1c48-kube-api-access-j2nfh\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" Oct 30 13:05:25.718538 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.718402 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-istio-token\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" Oct 30 13:05:25.718538 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.718425 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b656366-2609-4928-83f1-18bc76cb1c48-istio-data\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" Oct 30 13:05:25.718538 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.718450 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-credential-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" Oct 30 13:05:25.718538 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.718474 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b656366-2609-4928-83f1-18bc76cb1c48-istio-envoy\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" Oct 30 13:05:25.718538 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.718495 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-istiod-ca-cert\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" Oct 30 13:05:25.718538 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.718514 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-workload-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" Oct 30 13:05:25.718899 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.718589 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b656366-2609-4928-83f1-18bc76cb1c48-istiod-ca-cert\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" Oct 30 13:05:25.718899 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.718622 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-workload-certs\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" Oct 30 13:05:25.718899 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.718872 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-credential-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" Oct 30 13:05:25.719054 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.718919 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-workload-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" Oct 30 13:05:25.719109 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.719090 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-istio-data\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" Oct 30 13:05:25.719329 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.719307 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-istiod-ca-cert\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" Oct 30 13:05:25.719407 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.719381 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-istio-data\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" Oct 30 13:05:25.720622 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.720600 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-istio-envoy\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" Oct 30 13:05:25.720849 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.720832 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-istio-podinfo\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" Oct 30 13:05:25.747771 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.747743 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-istio-token\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" Oct 30 13:05:25.750519 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.750495 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fppzm\" (UniqueName: \"kubernetes.io/projected/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-kube-api-access-fppzm\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" Oct 30 13:05:25.796797 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.796760 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" Oct 30 13:05:25.802384 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:05:25.802348 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78d440f8_6cfb_4f8e_905e_228d07bdcec9.slice/crio-1b4504716a5489e3781f0be0a063fcf9f683ad26e91e19cd4181eff575521e63 WatchSource:0}: Error finding container 1b4504716a5489e3781f0be0a063fcf9f683ad26e91e19cd4181eff575521e63: Status 404 returned error can't find the container with id 1b4504716a5489e3781f0be0a063fcf9f683ad26e91e19cd4181eff575521e63 Oct 30 13:05:25.802958 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.802823 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv"] Oct 30 13:05:25.809989 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.809951 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:05:25.810076 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.810043 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:05:25.810119 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.810097 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:05:25.820877 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.820854 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b656366-2609-4928-83f1-18bc76cb1c48-credential-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" Oct 30 13:05:25.820972 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.820893 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2nfh\" (UniqueName: \"kubernetes.io/projected/0b656366-2609-4928-83f1-18bc76cb1c48-kube-api-access-j2nfh\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" Oct 30 13:05:25.821056 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.821034 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b656366-2609-4928-83f1-18bc76cb1c48-istio-data\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" Oct 30 13:05:25.821107 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.821095 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b656366-2609-4928-83f1-18bc76cb1c48-istio-envoy\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" Oct 30 13:05:25.821209 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.821192 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b656366-2609-4928-83f1-18bc76cb1c48-istiod-ca-cert\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" Oct 30 13:05:25.821324 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.821303 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b656366-2609-4928-83f1-18bc76cb1c48-credential-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" Oct 30 13:05:25.821430 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.821411 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b656366-2609-4928-83f1-18bc76cb1c48-istio-podinfo\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" Oct 30 13:05:25.821493 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.821438 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b656366-2609-4928-83f1-18bc76cb1c48-istio-data\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" Oct 30 13:05:25.821493 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.821444 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b656366-2609-4928-83f1-18bc76cb1c48-workload-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" Oct 30 13:05:25.821607 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.821503 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b656366-2609-4928-83f1-18bc76cb1c48-workload-certs\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" Oct 30 13:05:25.821607 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.821566 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b656366-2609-4928-83f1-18bc76cb1c48-istio-token\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" Oct 30 13:05:25.821989 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.821963 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b656366-2609-4928-83f1-18bc76cb1c48-istiod-ca-cert\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" Oct 30 13:05:25.822118 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.822097 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b656366-2609-4928-83f1-18bc76cb1c48-workload-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" Oct 30 13:05:25.822208 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.822180 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b656366-2609-4928-83f1-18bc76cb1c48-workload-certs\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" Oct 30 13:05:25.823827 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.823798 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b656366-2609-4928-83f1-18bc76cb1c48-istio-envoy\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" Oct 30 13:05:25.823943 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.823916 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b656366-2609-4928-83f1-18bc76cb1c48-istio-podinfo\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" Oct 30 13:05:25.850608 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.850581 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b656366-2609-4928-83f1-18bc76cb1c48-istio-token\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" Oct 30 13:05:25.853159 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.853131 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2nfh\" (UniqueName: \"kubernetes.io/projected/0b656366-2609-4928-83f1-18bc76cb1c48-kube-api-access-j2nfh\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" Oct 30 13:05:25.859774 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:25.857001 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" Oct 30 13:05:26.103705 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:26.103597 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" event={"ID":"78d440f8-6cfb-4f8e-905e-228d07bdcec9","Type":"ContainerStarted","Data":"d7c46df3328f7f5de418baaa55e2791ca3b2b0978746f0bc16e00a3130657e70"} Oct 30 13:05:26.103705 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:26.103656 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" event={"ID":"78d440f8-6cfb-4f8e-905e-228d07bdcec9","Type":"ContainerStarted","Data":"1b4504716a5489e3781f0be0a063fcf9f683ad26e91e19cd4181eff575521e63"} Oct 30 13:05:26.145408 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:05:26.145369 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c9814ec_a81d_4763_b0cb_97e20bccd5f7.slice/crio-a3957c8488dd67cfe511f929ecf84528d699996a8aa4aa074f51c0e3e689ef27 WatchSource:0}: Error finding container a3957c8488dd67cfe511f929ecf84528d699996a8aa4aa074f51c0e3e689ef27: Status 404 returned error can't find the container with id a3957c8488dd67cfe511f929ecf84528d699996a8aa4aa074f51c0e3e689ef27 Oct 30 13:05:26.145855 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:26.145835 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j"] Oct 30 13:05:26.148523 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:26.148490 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:05:26.148597 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:26.148554 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:05:26.148597 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:26.148582 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:05:26.173122 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:26.173078 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" podStartSLOduration=1.173063979 podStartE2EDuration="1.173063979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:26.169247654 +0000 UTC m=+1224.772746099" watchObservedRunningTime="2026-10-30 13:05:26.173063979 +0000 UTC m=+1224.776562425" Oct 30 13:05:26.200904 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:26.200873 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz"] Oct 30 13:05:26.203756 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:05:26.203724 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b656366_2609_4928_83f1_18bc76cb1c48.slice/crio-2b895f6308c638466fd610fcd44fc11c0cd6a9566999e8156cee882aebea5a8a WatchSource:0}: Error finding container 2b895f6308c638466fd610fcd44fc11c0cd6a9566999e8156cee882aebea5a8a: Status 404 returned error can't find the container with id 2b895f6308c638466fd610fcd44fc11c0cd6a9566999e8156cee882aebea5a8a Oct 30 13:05:26.207451 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:26.207417 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:05:26.207549 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:26.207484 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:05:26.207549 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:26.207517 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:05:26.462281 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:26.462244 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" Oct 30 13:05:26.463925 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:26.463895 4931 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 13:05:26.464078 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:26.463943 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" podUID="78d440f8-6cfb-4f8e-905e-228d07bdcec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 13:05:27.125210 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:27.125178 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" event={"ID":"0b656366-2609-4928-83f1-18bc76cb1c48","Type":"ContainerStarted","Data":"363b77e3a283332a48e1ff503ed6a0ba7ed19bb3ede1e220df2dbf83b1a4cdc7"} Oct 30 13:05:27.125498 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:27.125228 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" event={"ID":"0b656366-2609-4928-83f1-18bc76cb1c48","Type":"ContainerStarted","Data":"2b895f6308c638466fd610fcd44fc11c0cd6a9566999e8156cee882aebea5a8a"} Oct 30 13:05:27.127021 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:27.126933 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" event={"ID":"0c9814ec-a81d-4763-b0cb-97e20bccd5f7","Type":"ContainerStarted","Data":"42672481b26f80e5b9d75232519d3cd2b75cc0abb0440bbc4b5282f30f69dafb"} Oct 30 13:05:27.127021 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:27.126985 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" event={"ID":"0c9814ec-a81d-4763-b0cb-97e20bccd5f7","Type":"ContainerStarted","Data":"a3957c8488dd67cfe511f929ecf84528d699996a8aa4aa074f51c0e3e689ef27"} Oct 30 13:05:27.294292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:27.290663 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" podStartSLOduration=2.290638944 podStartE2EDuration="2.290638944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:27.217572393 +0000 UTC m=+1225.821070850" watchObservedRunningTime="2026-10-30 13:05:27.290638944 +0000 UTC m=+1225.894137392" Oct 30 13:05:27.311647 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:27.311544 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" podStartSLOduration=2.311517223 podStartE2EDuration="2.311517223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:27.295316808 +0000 UTC m=+1225.898815292" watchObservedRunningTime="2026-10-30 13:05:27.311517223 +0000 UTC m=+1225.915015670" Oct 30 13:05:27.472290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:27.466362 4931 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 13:05:27.472290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:27.466429 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" podUID="78d440f8-6cfb-4f8e-905e-228d07bdcec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 13:05:27.819620 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:27.819517 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" Oct 30 13:05:27.826127 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:27.826094 4931 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 13:05:27.826442 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:27.826416 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" podUID="0c9814ec-a81d-4763-b0cb-97e20bccd5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 13:05:27.891344 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:27.891307 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" Oct 30 13:05:27.891744 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:27.891647 4931 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 13:05:27.906435 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:27.906389 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" podUID="0b656366-2609-4928-83f1-18bc76cb1c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 13:05:28.465642 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:28.465604 4931 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 13:05:28.466288 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:28.466236 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" podUID="78d440f8-6cfb-4f8e-905e-228d07bdcec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 13:05:28.810357 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:28.800676 4931 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 13:05:28.810357 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:28.800759 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" podUID="0c9814ec-a81d-4763-b0cb-97e20bccd5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 13:05:28.860739 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:28.860690 4931 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 13:05:28.861101 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:28.861064 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" podUID="0b656366-2609-4928-83f1-18bc76cb1c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 13:05:29.473772 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:29.473707 4931 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 13:05:29.474427 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:29.474395 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" podUID="78d440f8-6cfb-4f8e-905e-228d07bdcec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 13:05:29.798040 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:29.797937 4931 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 13:05:29.798333 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:29.798297 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" podUID="0c9814ec-a81d-4763-b0cb-97e20bccd5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 13:05:29.857652 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:29.857614 4931 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 13:05:29.857969 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:29.857940 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" podUID="0b656366-2609-4928-83f1-18bc76cb1c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 13:05:30.464319 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:30.463610 4931 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 13:05:30.464319 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:30.463676 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" podUID="78d440f8-6cfb-4f8e-905e-228d07bdcec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 13:05:30.803348 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:30.800625 4931 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 13:05:30.803348 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:30.800688 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" podUID="0c9814ec-a81d-4763-b0cb-97e20bccd5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 13:05:30.864288 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:30.860295 4931 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 13:05:30.864288 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:30.860397 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" podUID="0b656366-2609-4928-83f1-18bc76cb1c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 13:05:31.501311 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:31.500768 4931 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 13:05:31.501311 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:31.500879 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" podUID="78d440f8-6cfb-4f8e-905e-228d07bdcec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 13:05:31.642934 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:31.642884 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" podUID="7ad1989a-c73f-4647-9e50-855605bdb08e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:31.728415 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:31.728360 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" podUID="089cb620-913d-47c7-b8f7-9b2298c194bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:31.747886 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:31.747836 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" podUID="e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:31.799848 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:31.799749 4931 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 13:05:31.800164 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:31.800128 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" podUID="0c9814ec-a81d-4763-b0cb-97e20bccd5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 13:05:31.864385 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:31.864337 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" podUID="d345fdd5-407b-4162-a2cd-64a40068cb3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:31.866525 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:31.866497 4931 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 13:05:31.866741 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:31.866719 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" podUID="0b656366-2609-4928-83f1-18bc76cb1c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 13:05:32.039003 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:32.038959 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" podUID="5a504237-e5a3-491f-8f84-0419cf502f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:32.186470 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:32.186361 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" podUID="64468005-61ea-4f2a-b72c-9c30c6a4d686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:32.220410 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:32.220362 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" podUID="f3b342db-9a01-46ad-b476-bfc912d9ce1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:32.467300 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:32.463541 4931 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 13:05:32.467300 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:32.463605 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" podUID="78d440f8-6cfb-4f8e-905e-228d07bdcec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 13:05:32.507305 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:32.498897 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" podUID="606b252b-881c-4d40-8ba9-fc4ed937a995" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 13:05:32.507305 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:32.499314 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" podUID="9577f571-30cd-4043-a3e4-72bf95837552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:32.646932 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:32.644380 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" podUID="7ad1989a-c73f-4647-9e50-855605bdb08e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 13:05:32.646932 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:32.645176 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:05:32.735471 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:32.735362 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" podUID="089cb620-913d-47c7-b8f7-9b2298c194bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:32.736488 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:32.736455 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 13:05:32.800821 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:32.800593 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" podUID="e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:32.804306 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:32.803295 4931 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 13:05:32.804306 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:32.803452 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" podUID="0c9814ec-a81d-4763-b0cb-97e20bccd5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 13:05:32.834327 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:32.809347 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:05:32.834327 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:32.811335 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:05:32.867595 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:32.867497 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" podUID="d345fdd5-407b-4162-a2cd-64a40068cb3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:32.873319 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:32.868534 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 13:05:33.041353 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:33.003436 4931 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 13:05:33.041353 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:33.003504 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" podUID="0b656366-2609-4928-83f1-18bc76cb1c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 13:05:33.082286 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:33.041902 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" podUID="5a504237-e5a3-491f-8f84-0419cf502f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:33.082286 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:33.043501 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:05:33.193336 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:33.193288 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" podUID="0c5a9f00-6d5b-4860-a210-2fec24afb322" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:33.194154 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:33.194113 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" podUID="64468005-61ea-4f2a-b72c-9c30c6a4d686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 13:05:33.204367 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:33.200288 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 13:05:33.269561 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:33.269490 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" podUID="f3b342db-9a01-46ad-b476-bfc912d9ce1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:33.270370 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:33.270348 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" Oct 30 13:05:33.464936 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:33.461446 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" podUID="b1e86eea-4824-4c50-8ad3-2a15c3365c3d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:33.571253 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:33.571207 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" podUID="606b252b-881c-4d40-8ba9-fc4ed937a995" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 13:05:33.591528 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:33.587548 4931 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 13:05:33.595289 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:33.591763 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" podUID="78d440f8-6cfb-4f8e-905e-228d07bdcec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 13:05:33.620375 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:33.594521 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" podUID="9577f571-30cd-4043-a3e4-72bf95837552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:33.704907 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:33.704873 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 13:05:33.707121 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:33.707096 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:05:33.749138 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:33.749037 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" podUID="089cb620-913d-47c7-b8f7-9b2298c194bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:33.830292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:33.805331 4931 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 13:05:33.830591 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:33.830556 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" podUID="0c9814ec-a81d-4763-b0cb-97e20bccd5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 13:05:33.830735 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:33.829421 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" podUID="e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 13:05:33.854630 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:33.828372 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" podUID="7ad1989a-c73f-4647-9e50-855605bdb08e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:33.929291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:33.882562 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" podUID="d345fdd5-407b-4162-a2cd-64a40068cb3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:33.956322 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:33.914412 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" podUID="65799c14-8de4-4b4a-90df-7cd257e1e581" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:33.956322 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:33.934059 4931 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 13:05:33.956322 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:33.934205 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" podUID="0b656366-2609-4928-83f1-18bc76cb1c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 13:05:34.044349 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:34.043361 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" podUID="5a504237-e5a3-491f-8f84-0419cf502f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:34.211318 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:34.200486 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" podUID="64468005-61ea-4f2a-b72c-9c30c6a4d686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:34.236290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:34.235690 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" podUID="0c5a9f00-6d5b-4860-a210-2fec24afb322" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:34.236489 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:34.236179 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 13:05:34.333972 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:34.333843 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-tvm7w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:05:34.334312 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:34.334255 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" podUID="3dd90613-12ac-4509-9f2e-b5c1e199719f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:34.370411 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:34.278258 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" podUID="f3b342db-9a01-46ad-b476-bfc912d9ce1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 30 13:05:34.502397 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:34.502214 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" podUID="b1e86eea-4824-4c50-8ad3-2a15c3365c3d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 13:05:34.507542 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:34.507507 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" Oct 30 13:05:34.535305 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:34.531227 4931 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 13:05:34.535305 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:34.531308 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" podUID="78d440f8-6cfb-4f8e-905e-228d07bdcec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 13:05:34.590556 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:34.590463 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" podUID="606b252b-881c-4d40-8ba9-fc4ed937a995" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:34.629684 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:34.629379 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" podUID="9577f571-30cd-4043-a3e4-72bf95837552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:34.665933 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:34.665783 4931 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:05:34.665933 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:34.665842 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" podUID="16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:34.749718 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:34.749599 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" podUID="089cb620-913d-47c7-b8f7-9b2298c194bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:34.824555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:34.820653 4931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" podUID="f3b342db-9a01-46ad-b476-bfc912d9ce1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:34.828301 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:34.825382 4931 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 13:05:34.828301 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:34.825447 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" podUID="0c9814ec-a81d-4763-b0cb-97e20bccd5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 13:05:34.892362 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:34.836622 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" podUID="e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:34.892362 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:34.854724 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" podUID="7ad1989a-c73f-4647-9e50-855605bdb08e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:34.892362 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:34.864473 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:05:34.892362 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:34.867093 4931 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 13:05:34.892362 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:34.867148 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" podUID="0b656366-2609-4928-83f1-18bc76cb1c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 13:05:34.933971 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:34.933923 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" podUID="65799c14-8de4-4b4a-90df-7cd257e1e581" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:34.934488 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:34.934443 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:05:35.052738 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:35.052694 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" podUID="5a504237-e5a3-491f-8f84-0419cf502f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:35.184605 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:35.088719 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" podUID="d345fdd5-407b-4162-a2cd-64a40068cb3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 13:05:35.405293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:35.395517 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" podUID="f3b342db-9a01-46ad-b476-bfc912d9ce1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:35.405293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:35.396460 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" podUID="0c5a9f00-6d5b-4860-a210-2fec24afb322" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 13:05:35.500322 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:35.479571 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" podUID="64468005-61ea-4f2a-b72c-9c30c6a4d686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 13:05:35.620325 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:35.585936 4931 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 13:05:35.621565 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:35.621518 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" podUID="78d440f8-6cfb-4f8e-905e-228d07bdcec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 13:05:35.660292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:35.634541 4931 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:05:35.660292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:35.634623 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" podUID="4e1ed676-cd86-4b17-97af-1a5a501f6274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:35.660292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:35.636130 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" podUID="606b252b-881c-4d40-8ba9-fc4ed937a995" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 13:05:35.682293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:35.663521 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" podUID="9577f571-30cd-4043-a3e4-72bf95837552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:35.954350 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:35.754567 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" podUID="b1e86eea-4824-4c50-8ad3-2a15c3365c3d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 13:05:35.973299 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:35.955443 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" podUID="089cb620-913d-47c7-b8f7-9b2298c194bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 13:05:35.998291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:35.974412 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" podUID="7ad1989a-c73f-4647-9e50-855605bdb08e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 13:05:36.016291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:36.001384 4931 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 13:05:36.016858 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:36.016744 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" podUID="0c9814ec-a81d-4763-b0cb-97e20bccd5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 13:05:36.046076 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:36.011902 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" podUID="65799c14-8de4-4b4a-90df-7cd257e1e581" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 13:05:36.046076 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:36.011820 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" podUID="e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:36.070479 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:36.067403 4931 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 13:05:36.070479 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:36.067487 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" podUID="0b656366-2609-4928-83f1-18bc76cb1c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 13:05:36.155933 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:36.155890 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" podUID="5a504237-e5a3-491f-8f84-0419cf502f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 13:05:36.228498 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:36.228377 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" podUID="d345fdd5-407b-4162-a2cd-64a40068cb3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:36.401294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:36.399349 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" podUID="0c5a9f00-6d5b-4860-a210-2fec24afb322" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:36.401294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:36.399736 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" podUID="f3b342db-9a01-46ad-b476-bfc912d9ce1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:36.503484 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:36.503375 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" podUID="64468005-61ea-4f2a-b72c-9c30c6a4d686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:36.651563 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:36.651521 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" podUID="606b252b-881c-4d40-8ba9-fc4ed937a995" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 13:05:36.683994 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:36.683952 4931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" podUID="5a504237-e5a3-491f-8f84-0419cf502f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded" Oct 30 13:05:36.684475 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:36.684450 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" podUID="9577f571-30cd-4043-a3e4-72bf95837552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 13:05:36.755477 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:36.755351 4931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" podUID="089cb620-913d-47c7-b8f7-9b2298c194bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:36.755757 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:36.755497 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" podUID="b1e86eea-4824-4c50-8ad3-2a15c3365c3d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:36.756241 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:36.755523 4931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" podUID="b1e86eea-4824-4c50-8ad3-2a15c3365c3d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": context deadline exceeded" Oct 30 13:05:36.798668 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:36.798631 4931 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 13:05:36.798905 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:36.798876 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" podUID="0c9814ec-a81d-4763-b0cb-97e20bccd5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 13:05:36.860771 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:36.860637 4931 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 13:05:36.860771 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:36.860700 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" podUID="0b656366-2609-4928-83f1-18bc76cb1c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 13:05:36.998978 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:36.998933 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" podUID="7ad1989a-c73f-4647-9e50-855605bdb08e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:37.038358 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:37.015566 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 13:05:37.038711 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:37.038676 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:05:37.094531 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:37.044619 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" podUID="089cb620-913d-47c7-b8f7-9b2298c194bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:37.131372 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:37.056417 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" podUID="65799c14-8de4-4b4a-90df-7cd257e1e581" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 13:05:37.131897 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:37.069308 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" podUID="e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:37.161885 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:37.161771 4931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" podUID="0c5a9f00-6d5b-4860-a210-2fec24afb322" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": context deadline exceeded" Oct 30 13:05:37.301465 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:37.301357 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" podUID="d345fdd5-407b-4162-a2cd-64a40068cb3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:37.406505 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:37.400692 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" podUID="f3b342db-9a01-46ad-b476-bfc912d9ce1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 30 13:05:37.406505 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:37.401050 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" podUID="0c5a9f00-6d5b-4860-a210-2fec24afb322" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 13:05:37.406505 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:37.403823 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" Oct 30 13:05:37.423476 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:37.423447 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" Oct 30 13:05:37.424032 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:37.424011 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:05:37.437231 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:37.437141 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:05:37.451734 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:37.451709 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:05:37.452790 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:37.452774 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" Oct 30 13:05:37.468930 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:37.453055 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" Oct 30 13:05:37.659213 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:37.659115 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" podUID="606b252b-881c-4d40-8ba9-fc4ed937a995" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:37.799398 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:05:37.794567 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:05:37.799608 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:37.796975 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" Oct 30 13:05:37.799681 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:37.799655 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 13:05:37.799741 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:37.799696 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 13:05:37.806293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:37.806241 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:05:37.932134 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:37.931460 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 13:05:37.932134 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:37.931520 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 13:05:38.090344 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:38.090312 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-775a7fbd-68rv9"] Oct 30 13:05:38.100770 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:38.100740 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-775a7fbd-68rv9"] Oct 30 13:05:38.370677 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:38.370644 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" Oct 30 13:05:38.371184 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:38.371161 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" Oct 30 13:05:38.384564 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:38.384538 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" Oct 30 13:05:38.385884 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:38.385726 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" Oct 30 13:05:38.386615 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:38.386570 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" Oct 30 13:05:38.387111 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:38.387073 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" Oct 30 13:05:39.385994 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:39.385959 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k"] Oct 30 13:05:39.391752 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:39.391644 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" Oct 30 13:05:39.411634 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:39.411611 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k"] Oct 30 13:05:39.417142 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:39.417111 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-7a3603\"/\"distinct-2-w5-7a3603-kserve-self-signed-certs\"" Oct 30 13:05:39.517112 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:39.517081 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="940c5766-5edb-44af-b989-7fe49f3b8e99" path="/var/lib/kubelet/pods/940c5766-5edb-44af-b989-7fe49f3b8e99/volumes" Oct 30 13:05:39.519350 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:39.519322 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee8505de-cac0-4e49-bd15-a044536b10e4-tmp-dir\") pod \"distinct-2-w5-7a3603-kserve-ff956599f-m5t5k\" (UID: \"ee8505de-cac0-4e49-bd15-a044536b10e4\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" Oct 30 13:05:39.519502 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:39.519375 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee8505de-cac0-4e49-bd15-a044536b10e4-dshm\") pod \"distinct-2-w5-7a3603-kserve-ff956599f-m5t5k\" (UID: \"ee8505de-cac0-4e49-bd15-a044536b10e4\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" Oct 30 13:05:39.519502 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:39.519466 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmvb4\" (UniqueName: \"kubernetes.io/projected/ee8505de-cac0-4e49-bd15-a044536b10e4-kube-api-access-nmvb4\") pod \"distinct-2-w5-7a3603-kserve-ff956599f-m5t5k\" (UID: \"ee8505de-cac0-4e49-bd15-a044536b10e4\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" Oct 30 13:05:39.519602 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:39.519579 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee8505de-cac0-4e49-bd15-a044536b10e4-model-cache\") pod \"distinct-2-w5-7a3603-kserve-ff956599f-m5t5k\" (UID: \"ee8505de-cac0-4e49-bd15-a044536b10e4\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" Oct 30 13:05:39.519745 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:39.519726 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee8505de-cac0-4e49-bd15-a044536b10e4-home\") pod \"distinct-2-w5-7a3603-kserve-ff956599f-m5t5k\" (UID: \"ee8505de-cac0-4e49-bd15-a044536b10e4\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" Oct 30 13:05:39.519861 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:39.519844 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee8505de-cac0-4e49-bd15-a044536b10e4-tls-certs\") pod \"distinct-2-w5-7a3603-kserve-ff956599f-m5t5k\" (UID: \"ee8505de-cac0-4e49-bd15-a044536b10e4\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" Oct 30 13:05:39.620939 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:39.620906 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee8505de-cac0-4e49-bd15-a044536b10e4-tmp-dir\") pod \"distinct-2-w5-7a3603-kserve-ff956599f-m5t5k\" (UID: \"ee8505de-cac0-4e49-bd15-a044536b10e4\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" Oct 30 13:05:39.621131 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:39.620973 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee8505de-cac0-4e49-bd15-a044536b10e4-dshm\") pod \"distinct-2-w5-7a3603-kserve-ff956599f-m5t5k\" (UID: \"ee8505de-cac0-4e49-bd15-a044536b10e4\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" Oct 30 13:05:39.621131 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:39.621018 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmvb4\" (UniqueName: \"kubernetes.io/projected/ee8505de-cac0-4e49-bd15-a044536b10e4-kube-api-access-nmvb4\") pod \"distinct-2-w5-7a3603-kserve-ff956599f-m5t5k\" (UID: \"ee8505de-cac0-4e49-bd15-a044536b10e4\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" Oct 30 13:05:39.621131 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:39.621054 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee8505de-cac0-4e49-bd15-a044536b10e4-model-cache\") pod \"distinct-2-w5-7a3603-kserve-ff956599f-m5t5k\" (UID: \"ee8505de-cac0-4e49-bd15-a044536b10e4\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" Oct 30 13:05:39.621334 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:39.621158 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee8505de-cac0-4e49-bd15-a044536b10e4-home\") pod \"distinct-2-w5-7a3603-kserve-ff956599f-m5t5k\" (UID: \"ee8505de-cac0-4e49-bd15-a044536b10e4\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" Oct 30 13:05:39.621334 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:39.621310 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee8505de-cac0-4e49-bd15-a044536b10e4-tls-certs\") pod \"distinct-2-w5-7a3603-kserve-ff956599f-m5t5k\" (UID: \"ee8505de-cac0-4e49-bd15-a044536b10e4\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" Oct 30 13:05:39.621453 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:39.621396 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee8505de-cac0-4e49-bd15-a044536b10e4-tmp-dir\") pod \"distinct-2-w5-7a3603-kserve-ff956599f-m5t5k\" (UID: \"ee8505de-cac0-4e49-bd15-a044536b10e4\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" Oct 30 13:05:39.621713 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:39.621689 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee8505de-cac0-4e49-bd15-a044536b10e4-model-cache\") pod \"distinct-2-w5-7a3603-kserve-ff956599f-m5t5k\" (UID: \"ee8505de-cac0-4e49-bd15-a044536b10e4\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" Oct 30 13:05:39.621824 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:39.621715 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee8505de-cac0-4e49-bd15-a044536b10e4-home\") pod \"distinct-2-w5-7a3603-kserve-ff956599f-m5t5k\" (UID: \"ee8505de-cac0-4e49-bd15-a044536b10e4\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" Oct 30 13:05:39.623490 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:39.623460 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee8505de-cac0-4e49-bd15-a044536b10e4-dshm\") pod \"distinct-2-w5-7a3603-kserve-ff956599f-m5t5k\" (UID: \"ee8505de-cac0-4e49-bd15-a044536b10e4\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" Oct 30 13:05:39.623914 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:39.623893 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee8505de-cac0-4e49-bd15-a044536b10e4-tls-certs\") pod \"distinct-2-w5-7a3603-kserve-ff956599f-m5t5k\" (UID: \"ee8505de-cac0-4e49-bd15-a044536b10e4\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" Oct 30 13:05:39.672624 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:39.672541 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmvb4\" (UniqueName: \"kubernetes.io/projected/ee8505de-cac0-4e49-bd15-a044536b10e4-kube-api-access-nmvb4\") pod \"distinct-2-w5-7a3603-kserve-ff956599f-m5t5k\" (UID: \"ee8505de-cac0-4e49-bd15-a044536b10e4\") " pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" Oct 30 13:05:39.710802 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:39.710758 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" Oct 30 13:05:40.088164 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:40.088129 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k"] Oct 30 13:05:40.848695 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:40.848654 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" event={"ID":"ee8505de-cac0-4e49-bd15-a044536b10e4","Type":"ContainerStarted","Data":"14cab76f1504906ee252c3f7c56a7f92b85f76dab5bed5fbf0bc6dc443be0ab9"} Oct 30 13:05:40.849129 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:40.848709 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" event={"ID":"ee8505de-cac0-4e49-bd15-a044536b10e4","Type":"ContainerStarted","Data":"b5ffa2b94e368885f63f625aed37e31a8f6518c5c6526887b27f0886935a0b14"} Oct 30 13:05:41.711467 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:41.711432 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" Oct 30 13:05:49.711842 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:49.711804 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" Oct 30 13:05:49.724390 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:49.724362 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" Oct 30 13:05:49.775393 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:49.775319 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" podStartSLOduration=10.775301302999999 podStartE2EDuration="10.775301303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:40.913580974 +0000 UTC m=+1239.517079420" watchObservedRunningTime="2026-10-30 13:05:49.775301303 +0000 UTC m=+1248.378799749" Oct 30 13:05:49.934617 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:49.934590 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" Oct 30 13:05:50.920027 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:50.919990 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68573cef-5wsvd"] Oct 30 13:05:50.924233 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:50.924212 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68573cef-5wsvd" Oct 30 13:05:50.950988 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:50.950929 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68573cef-5wsvd"] Oct 30 13:05:50.955711 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:50.955687 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:05:50.955711 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:50.955702 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:05:50.988392 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:50.988354 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71b78d29-7835-45f9-bf01-9a8c87906aff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-68573cef-5wsvd\" (UID: \"71b78d29-7835-45f9-bf01-9a8c87906aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68573cef-5wsvd" Oct 30 13:05:50.988584 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:50.988491 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71b78d29-7835-45f9-bf01-9a8c87906aff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-68573cef-5wsvd\" (UID: \"71b78d29-7835-45f9-bf01-9a8c87906aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68573cef-5wsvd" Oct 30 13:05:50.988656 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:50.988618 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdl76\" (UniqueName: \"kubernetes.io/projected/71b78d29-7835-45f9-bf01-9a8c87906aff-kube-api-access-hdl76\") pod \"maas-api-revoke-keys-e2e-ait-68573cef-5wsvd\" (UID: \"71b78d29-7835-45f9-bf01-9a8c87906aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68573cef-5wsvd" Oct 30 13:05:51.089972 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:51.089940 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71b78d29-7835-45f9-bf01-9a8c87906aff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-68573cef-5wsvd\" (UID: \"71b78d29-7835-45f9-bf01-9a8c87906aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68573cef-5wsvd" Oct 30 13:05:51.090142 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:51.090015 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71b78d29-7835-45f9-bf01-9a8c87906aff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-68573cef-5wsvd\" (UID: \"71b78d29-7835-45f9-bf01-9a8c87906aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68573cef-5wsvd" Oct 30 13:05:51.090142 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:51.090081 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdl76\" (UniqueName: \"kubernetes.io/projected/71b78d29-7835-45f9-bf01-9a8c87906aff-kube-api-access-hdl76\") pod \"maas-api-revoke-keys-e2e-ait-68573cef-5wsvd\" (UID: \"71b78d29-7835-45f9-bf01-9a8c87906aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68573cef-5wsvd" Oct 30 13:05:51.090669 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:51.090646 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71b78d29-7835-45f9-bf01-9a8c87906aff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-68573cef-5wsvd\" (UID: \"71b78d29-7835-45f9-bf01-9a8c87906aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68573cef-5wsvd" Oct 30 13:05:51.092432 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:51.092411 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71b78d29-7835-45f9-bf01-9a8c87906aff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-68573cef-5wsvd\" (UID: \"71b78d29-7835-45f9-bf01-9a8c87906aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68573cef-5wsvd" Oct 30 13:05:51.114856 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:51.114831 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdl76\" (UniqueName: \"kubernetes.io/projected/71b78d29-7835-45f9-bf01-9a8c87906aff-kube-api-access-hdl76\") pod \"maas-api-revoke-keys-e2e-ait-68573cef-5wsvd\" (UID: \"71b78d29-7835-45f9-bf01-9a8c87906aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68573cef-5wsvd" Oct 30 13:05:51.240138 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:51.240105 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68573cef-5wsvd" Oct 30 13:05:51.566094 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:51.566057 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68573cef-5wsvd"] Oct 30 13:05:51.566477 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:05:51.566447 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b78d29_7835_45f9_bf01_9a8c87906aff.slice/crio-fbf339e55fdc61a39c17e33ee18f5febc34b379526f12e84fcbcac1d6bd2ddae WatchSource:0}: Error finding container fbf339e55fdc61a39c17e33ee18f5febc34b379526f12e84fcbcac1d6bd2ddae: Status 404 returned error can't find the container with id fbf339e55fdc61a39c17e33ee18f5febc34b379526f12e84fcbcac1d6bd2ddae Oct 30 13:05:51.942422 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:51.942382 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68573cef-5wsvd" event={"ID":"71b78d29-7835-45f9-bf01-9a8c87906aff","Type":"ContainerStarted","Data":"a8a97a8ab1dad74142ecc6acd13bb7f10e22c503f0bc43f36589afde4c018c79"} Oct 30 13:05:51.942839 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:51.942443 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68573cef-5wsvd" event={"ID":"71b78d29-7835-45f9-bf01-9a8c87906aff","Type":"ContainerStarted","Data":"fbf339e55fdc61a39c17e33ee18f5febc34b379526f12e84fcbcac1d6bd2ddae"} Oct 30 13:05:51.997799 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:51.997742 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68573cef-5wsvd" podStartSLOduration=1.997726767 podStartE2EDuration="1.997726767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:51.985239267 +0000 UTC m=+1250.588737713" watchObservedRunningTime="2026-10-30 13:05:51.997726767 +0000 UTC m=+1250.601225234" Oct 30 13:05:52.951595 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:52.951563 4931 generic.go:356] "Generic (PLEG): container finished" podID="71b78d29-7835-45f9-bf01-9a8c87906aff" containerID="a8a97a8ab1dad74142ecc6acd13bb7f10e22c503f0bc43f36589afde4c018c79" exitCode=0 Oct 30 13:05:52.951996 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:52.951600 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68573cef-5wsvd" event={"ID":"71b78d29-7835-45f9-bf01-9a8c87906aff","Type":"ContainerDied","Data":"a8a97a8ab1dad74142ecc6acd13bb7f10e22c503f0bc43f36589afde4c018c79"} Oct 30 13:05:53.035313 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:53.035261 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac033438-rhwjd"] Oct 30 13:05:53.042976 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:53.042940 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac033438-rhwjd"] Oct 30 13:05:53.131788 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:53.131759 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-752d3a58-8ff6n"] Oct 30 13:05:53.135989 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:53.135973 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-752d3a58-8ff6n" Oct 30 13:05:53.156981 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:53.156944 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-752d3a58-8ff6n"] Oct 30 13:05:53.221479 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:53.221396 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-752d3a58-8ff6n\" (UID: \"ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-752d3a58-8ff6n" Oct 30 13:05:53.221479 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:53.221460 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-752d3a58-8ff6n\" (UID: \"ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-752d3a58-8ff6n" Oct 30 13:05:53.221684 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:53.221556 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sr6t\" (UniqueName: \"kubernetes.io/projected/ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe-kube-api-access-2sr6t\") pod \"maas-api-revoke-keys-e2e-ait-752d3a58-8ff6n\" (UID: \"ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-752d3a58-8ff6n" Oct 30 13:05:53.323064 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:53.323023 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-752d3a58-8ff6n\" (UID: \"ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-752d3a58-8ff6n" Oct 30 13:05:53.323284 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:53.323115 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-752d3a58-8ff6n\" (UID: \"ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-752d3a58-8ff6n" Oct 30 13:05:53.323284 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:53.323172 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sr6t\" (UniqueName: \"kubernetes.io/projected/ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe-kube-api-access-2sr6t\") pod \"maas-api-revoke-keys-e2e-ait-752d3a58-8ff6n\" (UID: \"ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-752d3a58-8ff6n" Oct 30 13:05:53.323877 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:53.323850 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-752d3a58-8ff6n\" (UID: \"ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-752d3a58-8ff6n" Oct 30 13:05:53.325482 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:53.325456 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-752d3a58-8ff6n\" (UID: \"ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-752d3a58-8ff6n" Oct 30 13:05:53.348702 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:53.348675 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sr6t\" (UniqueName: \"kubernetes.io/projected/ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe-kube-api-access-2sr6t\") pod \"maas-api-revoke-keys-e2e-ait-752d3a58-8ff6n\" (UID: \"ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-752d3a58-8ff6n" Oct 30 13:05:53.450602 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:53.450571 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-752d3a58-8ff6n" Oct 30 13:05:53.520732 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:53.520702 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06e41c6f-53ac-4719-b4f2-7631d6acbbef" path="/var/lib/kubelet/pods/06e41c6f-53ac-4719-b4f2-7631d6acbbef/volumes" Oct 30 13:05:53.771032 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:05:53.771000 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec8fcdd9_8f5b_4954_ad28_a8141acf0fbe.slice/crio-12f62cf56a04bbbc53a03c923eb5be1ac60a6bc9f79b0751d28715bc2aca8851 WatchSource:0}: Error finding container 12f62cf56a04bbbc53a03c923eb5be1ac60a6bc9f79b0751d28715bc2aca8851: Status 404 returned error can't find the container with id 12f62cf56a04bbbc53a03c923eb5be1ac60a6bc9f79b0751d28715bc2aca8851 Oct 30 13:05:53.771839 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:53.771815 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-752d3a58-8ff6n"] Oct 30 13:05:53.960203 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:53.960118 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-752d3a58-8ff6n" event={"ID":"ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe","Type":"ContainerStarted","Data":"12f62cf56a04bbbc53a03c923eb5be1ac60a6bc9f79b0751d28715bc2aca8851"} Oct 30 13:05:54.110915 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:54.110885 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68573cef-5wsvd" Oct 30 13:05:54.235822 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:54.235785 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71b78d29-7835-45f9-bf01-9a8c87906aff-kube-api-access-hdl76\" (UniqueName: \"kubernetes.io/projected/71b78d29-7835-45f9-bf01-9a8c87906aff-kube-api-access-hdl76\") pod \"71b78d29-7835-45f9-bf01-9a8c87906aff\" (UID: \"71b78d29-7835-45f9-bf01-9a8c87906aff\") " Oct 30 13:05:54.236032 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:54.235833 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71b78d29-7835-45f9-bf01-9a8c87906aff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71b78d29-7835-45f9-bf01-9a8c87906aff-maas-api-xks-ca\") pod \"71b78d29-7835-45f9-bf01-9a8c87906aff\" (UID: \"71b78d29-7835-45f9-bf01-9a8c87906aff\") " Oct 30 13:05:54.236919 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:54.236112 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71b78d29-7835-45f9-bf01-9a8c87906aff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71b78d29-7835-45f9-bf01-9a8c87906aff-maas-api-openshift-ca\") pod \"71b78d29-7835-45f9-bf01-9a8c87906aff\" (UID: \"71b78d29-7835-45f9-bf01-9a8c87906aff\") " Oct 30 13:05:54.236919 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:54.236456 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b78d29-7835-45f9-bf01-9a8c87906aff-maas-api-openshift-ca" pod "71b78d29-7835-45f9-bf01-9a8c87906aff" (UID: "71b78d29-7835-45f9-bf01-9a8c87906aff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:54.236919 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:54.236886 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71b78d29-7835-45f9-bf01-9a8c87906aff-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:05:54.237956 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:54.237926 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71b78d29-7835-45f9-bf01-9a8c87906aff-maas-api-xks-ca" pod "71b78d29-7835-45f9-bf01-9a8c87906aff" (UID: "71b78d29-7835-45f9-bf01-9a8c87906aff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:54.238080 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:54.238055 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b78d29-7835-45f9-bf01-9a8c87906aff-kube-api-access-hdl76" pod "71b78d29-7835-45f9-bf01-9a8c87906aff" (UID: "71b78d29-7835-45f9-bf01-9a8c87906aff"). InnerVolumeSpecName "kube-api-access-hdl76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:54.338717 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:54.338565 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdl76\" (UniqueName: \"kubernetes.io/projected/71b78d29-7835-45f9-bf01-9a8c87906aff-kube-api-access-hdl76\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:05:54.338717 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:54.338591 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71b78d29-7835-45f9-bf01-9a8c87906aff-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:05:54.972457 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:54.972428 4931 generic.go:356] "Generic (PLEG): container finished" podID="ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe" containerID="f1362f87bb51958f3c25195c62dd62a1d1e9c36969afa002ba31d7b2c50151dd" exitCode=0 Oct 30 13:05:54.972869 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:54.972516 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-752d3a58-8ff6n" event={"ID":"ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe","Type":"ContainerDied","Data":"f1362f87bb51958f3c25195c62dd62a1d1e9c36969afa002ba31d7b2c50151dd"} Oct 30 13:05:54.974009 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:54.973990 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68573cef-5wsvd" Oct 30 13:05:54.974129 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:54.973992 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68573cef-5wsvd" event={"ID":"71b78d29-7835-45f9-bf01-9a8c87906aff","Type":"ContainerDied","Data":"fbf339e55fdc61a39c17e33ee18f5febc34b379526f12e84fcbcac1d6bd2ddae"} Oct 30 13:05:54.974172 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:54.974126 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbf339e55fdc61a39c17e33ee18f5febc34b379526f12e84fcbcac1d6bd2ddae" Oct 30 13:05:55.894284 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:55.889943 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:05:56.172213 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:56.172184 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-752d3a58-8ff6n" Oct 30 13:05:56.262703 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:56.262659 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe-kube-api-access-2sr6t\" (UniqueName: \"kubernetes.io/projected/ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe-kube-api-access-2sr6t\") pod \"ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe\" (UID: \"ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe\") " Oct 30 13:05:56.262897 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:56.262792 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe-maas-api-xks-ca\") pod \"ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe\" (UID: \"ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe\") " Oct 30 13:05:56.263126 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:56.263099 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe-maas-api-openshift-ca\") pod \"ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe\" (UID: \"ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe\") " Oct 30 13:05:56.263538 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:56.263511 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe-maas-api-openshift-ca" pod "ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe" (UID: "ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:56.263977 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:56.263955 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:05:56.265162 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:56.265121 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe-maas-api-xks-ca" pod "ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe" (UID: "ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:56.265435 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:56.265407 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe-kube-api-access-2sr6t" pod "ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe" (UID: "ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe"). InnerVolumeSpecName "kube-api-access-2sr6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:56.366010 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:56.365969 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sr6t\" (UniqueName: \"kubernetes.io/projected/ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe-kube-api-access-2sr6t\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:05:56.366010 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:56.366003 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:05:56.999859 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:56.999823 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-752d3a58-8ff6n" Oct 30 13:05:57.000037 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:56.999828 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-752d3a58-8ff6n" event={"ID":"ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe","Type":"ContainerDied","Data":"12f62cf56a04bbbc53a03c923eb5be1ac60a6bc9f79b0751d28715bc2aca8851"} Oct 30 13:05:57.000037 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:57.000027 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12f62cf56a04bbbc53a03c923eb5be1ac60a6bc9f79b0751d28715bc2aca8851" Oct 30 13:05:58.669868 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:58.669818 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" podUID="606b252b-881c-4d40-8ba9-fc4ed937a995" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:58.685132 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:58.685013 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" podUID="9577f571-30cd-4043-a3e4-72bf95837552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 13:05:58.768458 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:58.768405 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" podUID="b1e86eea-4824-4c50-8ad3-2a15c3365c3d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:59.011459 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:59.011402 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" podUID="7ad1989a-c73f-4647-9e50-855605bdb08e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:59.209458 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:59.209397 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" podUID="64468005-61ea-4f2a-b72c-9c30c6a4d686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:59.230367 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:05:59.230330 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" podUID="d345fdd5-407b-4162-a2cd-64a40068cb3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:06:04.038466 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:04.038429 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-479d95c1-thznf"] Oct 30 13:06:04.051359 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:04.051326 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-479d95c1-thznf"] Oct 30 13:06:05.043910 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:05.043875 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59337106-4z5sd"] Oct 30 13:06:05.057809 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:05.057777 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59337106-4z5sd"] Oct 30 13:06:05.471635 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:05.471600 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68573cef-5wsvd"] Oct 30 13:06:05.488406 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:05.488374 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68573cef-5wsvd"] Oct 30 13:06:05.513966 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:05.513934 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="267c535f-c034-4727-8c4f-55c6565e5ffb" path="/var/lib/kubelet/pods/267c535f-c034-4727-8c4f-55c6565e5ffb/volumes" Oct 30 13:06:05.514448 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:05.514432 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d8a2bcf-c3d1-4370-a603-1728b9958a5a" path="/var/lib/kubelet/pods/5d8a2bcf-c3d1-4370-a603-1728b9958a5a/volumes" Oct 30 13:06:05.514844 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:05.514830 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71b78d29-7835-45f9-bf01-9a8c87906aff" path="/var/lib/kubelet/pods/71b78d29-7835-45f9-bf01-9a8c87906aff/volumes" Oct 30 13:06:07.354994 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:06:07.354949 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:06:08.393539 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:08.393197 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-752d3a58-8ff6n"] Oct 30 13:06:08.406030 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:08.405965 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-752d3a58-8ff6n"] Oct 30 13:06:09.514720 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:09.514688 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe" path="/var/lib/kubelet/pods/ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe/volumes" Oct 30 13:06:15.036572 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:15.036534 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7983812e-nqpd8"] Oct 30 13:06:15.050083 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:15.050054 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7983812e-nqpd8"] Oct 30 13:06:15.514200 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:15.514161 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f69c855f-3d94-41af-baec-6fb3190c4f75" path="/var/lib/kubelet/pods/f69c855f-3d94-41af-baec-6fb3190c4f75/volumes" Oct 30 13:06:24.698803 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:24.698768 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:06:29.109921 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:29.109877 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02b7cd66-7587p"] Oct 30 13:06:29.110533 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:29.110517 4931 cpu_manager.go:414] "containerMap: removing container" podUID="ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe" containerName="revoke-keys" Oct 30 13:06:29.110580 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:29.110535 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe" containerName="revoke-keys" Oct 30 13:06:29.110580 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:29.110556 4931 cpu_manager.go:414] "containerMap: removing container" podUID="71b78d29-7835-45f9-bf01-9a8c87906aff" containerName="revoke-keys" Oct 30 13:06:29.110580 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:29.110560 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71b78d29-7835-45f9-bf01-9a8c87906aff" containerName="revoke-keys" Oct 30 13:06:29.110685 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:29.110662 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="71b78d29-7835-45f9-bf01-9a8c87906aff" containerName="revoke-keys" Oct 30 13:06:29.110685 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:29.110673 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec8fcdd9-8f5b-4954-ad28-a8141acf0fbe" containerName="revoke-keys" Oct 30 13:06:29.114548 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:29.114527 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b7cd66-7587p" Oct 30 13:06:29.134538 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:29.134502 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02b7cd66-7587p"] Oct 30 13:06:29.142595 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:29.142564 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:06:29.142775 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:29.142630 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:06:29.243945 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:29.243909 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d706c855-11e7-443a-9dbf-00b9577f5b9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02b7cd66-7587p\" (UID: \"d706c855-11e7-443a-9dbf-00b9577f5b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b7cd66-7587p" Oct 30 13:06:29.244148 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:29.243988 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqs2n\" (UniqueName: \"kubernetes.io/projected/d706c855-11e7-443a-9dbf-00b9577f5b9e-kube-api-access-wqs2n\") pod \"maas-api-revoke-sub-02b7cd66-7587p\" (UID: \"d706c855-11e7-443a-9dbf-00b9577f5b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b7cd66-7587p" Oct 30 13:06:29.244148 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:29.244090 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d706c855-11e7-443a-9dbf-00b9577f5b9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02b7cd66-7587p\" (UID: \"d706c855-11e7-443a-9dbf-00b9577f5b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b7cd66-7587p" Oct 30 13:06:29.346140 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:29.346102 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d706c855-11e7-443a-9dbf-00b9577f5b9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02b7cd66-7587p\" (UID: \"d706c855-11e7-443a-9dbf-00b9577f5b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b7cd66-7587p" Oct 30 13:06:29.346387 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:29.346184 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqs2n\" (UniqueName: \"kubernetes.io/projected/d706c855-11e7-443a-9dbf-00b9577f5b9e-kube-api-access-wqs2n\") pod \"maas-api-revoke-sub-02b7cd66-7587p\" (UID: \"d706c855-11e7-443a-9dbf-00b9577f5b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b7cd66-7587p" Oct 30 13:06:29.346387 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:29.346239 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d706c855-11e7-443a-9dbf-00b9577f5b9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02b7cd66-7587p\" (UID: \"d706c855-11e7-443a-9dbf-00b9577f5b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b7cd66-7587p" Oct 30 13:06:29.346876 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:29.346853 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d706c855-11e7-443a-9dbf-00b9577f5b9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02b7cd66-7587p\" (UID: \"d706c855-11e7-443a-9dbf-00b9577f5b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b7cd66-7587p" Oct 30 13:06:29.348717 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:29.348694 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d706c855-11e7-443a-9dbf-00b9577f5b9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02b7cd66-7587p\" (UID: \"d706c855-11e7-443a-9dbf-00b9577f5b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b7cd66-7587p" Oct 30 13:06:29.370840 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:29.370783 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqs2n\" (UniqueName: \"kubernetes.io/projected/d706c855-11e7-443a-9dbf-00b9577f5b9e-kube-api-access-wqs2n\") pod \"maas-api-revoke-sub-02b7cd66-7587p\" (UID: \"d706c855-11e7-443a-9dbf-00b9577f5b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b7cd66-7587p" Oct 30 13:06:29.434636 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:29.434610 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b7cd66-7587p" Oct 30 13:06:29.744054 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:29.744022 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02b7cd66-7587p"] Oct 30 13:06:29.744206 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:06:29.744186 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd706c855_11e7_443a_9dbf_00b9577f5b9e.slice/crio-b5e79af5fce576930cc0d5888e3cb3a9646fac0987b02206a24239a1a485178e WatchSource:0}: Error finding container b5e79af5fce576930cc0d5888e3cb3a9646fac0987b02206a24239a1a485178e: Status 404 returned error can't find the container with id b5e79af5fce576930cc0d5888e3cb3a9646fac0987b02206a24239a1a485178e Oct 30 13:06:30.451954 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:30.451910 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b7cd66-7587p" event={"ID":"d706c855-11e7-443a-9dbf-00b9577f5b9e","Type":"ContainerStarted","Data":"62e88201f1dabe6d3d89e41f39edecc2bcc3cf8277045bd8be286120387762ed"} Oct 30 13:06:30.452457 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:30.451998 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b7cd66-7587p" event={"ID":"d706c855-11e7-443a-9dbf-00b9577f5b9e","Type":"ContainerStarted","Data":"b5e79af5fce576930cc0d5888e3cb3a9646fac0987b02206a24239a1a485178e"} Oct 30 13:06:30.496302 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:30.496228 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b7cd66-7587p" podStartSLOduration=1.4962139859999999 podStartE2EDuration="1.496213986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:30.494528978 +0000 UTC m=+1289.098027429" watchObservedRunningTime="2026-10-30 13:06:30.496213986 +0000 UTC m=+1289.099712462" Oct 30 13:06:31.461802 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:31.461769 4931 generic.go:356] "Generic (PLEG): container finished" podID="d706c855-11e7-443a-9dbf-00b9577f5b9e" containerID="62e88201f1dabe6d3d89e41f39edecc2bcc3cf8277045bd8be286120387762ed" exitCode=0 Oct 30 13:06:31.462200 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:31.461851 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b7cd66-7587p" event={"ID":"d706c855-11e7-443a-9dbf-00b9577f5b9e","Type":"ContainerDied","Data":"62e88201f1dabe6d3d89e41f39edecc2bcc3cf8277045bd8be286120387762ed"} Oct 30 13:06:32.639086 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:32.639059 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b7cd66-7587p" Oct 30 13:06:32.698121 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:32.698088 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d706c855-11e7-443a-9dbf-00b9577f5b9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d706c855-11e7-443a-9dbf-00b9577f5b9e-maas-api-xks-ca\") pod \"d706c855-11e7-443a-9dbf-00b9577f5b9e\" (UID: \"d706c855-11e7-443a-9dbf-00b9577f5b9e\") " Oct 30 13:06:32.698338 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:32.698142 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d706c855-11e7-443a-9dbf-00b9577f5b9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d706c855-11e7-443a-9dbf-00b9577f5b9e-maas-api-openshift-ca\") pod \"d706c855-11e7-443a-9dbf-00b9577f5b9e\" (UID: \"d706c855-11e7-443a-9dbf-00b9577f5b9e\") " Oct 30 13:06:32.698338 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:32.698291 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d706c855-11e7-443a-9dbf-00b9577f5b9e-kube-api-access-wqs2n\" (UniqueName: \"kubernetes.io/projected/d706c855-11e7-443a-9dbf-00b9577f5b9e-kube-api-access-wqs2n\") pod \"d706c855-11e7-443a-9dbf-00b9577f5b9e\" (UID: \"d706c855-11e7-443a-9dbf-00b9577f5b9e\") " Oct 30 13:06:32.698548 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:32.698519 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d706c855-11e7-443a-9dbf-00b9577f5b9e-maas-api-openshift-ca" pod "d706c855-11e7-443a-9dbf-00b9577f5b9e" (UID: "d706c855-11e7-443a-9dbf-00b9577f5b9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:32.699062 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:32.699037 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d706c855-11e7-443a-9dbf-00b9577f5b9e-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:06:32.700337 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:32.700312 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d706c855-11e7-443a-9dbf-00b9577f5b9e-maas-api-xks-ca" pod "d706c855-11e7-443a-9dbf-00b9577f5b9e" (UID: "d706c855-11e7-443a-9dbf-00b9577f5b9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:32.700588 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:32.700560 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d706c855-11e7-443a-9dbf-00b9577f5b9e-kube-api-access-wqs2n" pod "d706c855-11e7-443a-9dbf-00b9577f5b9e" (UID: "d706c855-11e7-443a-9dbf-00b9577f5b9e"). InnerVolumeSpecName "kube-api-access-wqs2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:32.800642 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:32.800553 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqs2n\" (UniqueName: \"kubernetes.io/projected/d706c855-11e7-443a-9dbf-00b9577f5b9e-kube-api-access-wqs2n\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:06:32.800642 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:32.800590 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d706c855-11e7-443a-9dbf-00b9577f5b9e-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:06:33.478113 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:33.478080 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b7cd66-7587p" Oct 30 13:06:33.478307 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:33.478105 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b7cd66-7587p" event={"ID":"d706c855-11e7-443a-9dbf-00b9577f5b9e","Type":"ContainerDied","Data":"b5e79af5fce576930cc0d5888e3cb3a9646fac0987b02206a24239a1a485178e"} Oct 30 13:06:33.478307 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:33.478157 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5e79af5fce576930cc0d5888e3cb3a9646fac0987b02206a24239a1a485178e" Oct 30 13:06:37.353769 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:06:37.353732 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:06:39.305182 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:39.305145 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-015dc049-clz92"] Oct 30 13:06:39.305787 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:39.305773 4931 cpu_manager.go:414] "containerMap: removing container" podUID="d706c855-11e7-443a-9dbf-00b9577f5b9e" containerName="revoke-keys" Oct 30 13:06:39.305787 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:39.305787 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d706c855-11e7-443a-9dbf-00b9577f5b9e" containerName="revoke-keys" Oct 30 13:06:39.305914 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:39.305890 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="d706c855-11e7-443a-9dbf-00b9577f5b9e" containerName="revoke-keys" Oct 30 13:06:39.309578 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:39.309561 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015dc049-clz92" Oct 30 13:06:39.330462 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:39.330433 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-015dc049-clz92"] Oct 30 13:06:39.378832 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:39.378799 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:06:39.378989 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:39.378884 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:06:39.394493 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:39.394465 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ca8cd79-0cc2-4813-aa69-e656ab2a1718-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-015dc049-clz92\" (UID: \"3ca8cd79-0cc2-4813-aa69-e656ab2a1718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015dc049-clz92" Oct 30 13:06:39.394634 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:39.394543 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ca8cd79-0cc2-4813-aa69-e656ab2a1718-maas-api-xks-ca\") pod \"maas-api-revoke-sub-015dc049-clz92\" (UID: \"3ca8cd79-0cc2-4813-aa69-e656ab2a1718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015dc049-clz92" Oct 30 13:06:39.394683 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:39.394663 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb559\" (UniqueName: \"kubernetes.io/projected/3ca8cd79-0cc2-4813-aa69-e656ab2a1718-kube-api-access-rb559\") pod \"maas-api-revoke-sub-015dc049-clz92\" (UID: \"3ca8cd79-0cc2-4813-aa69-e656ab2a1718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015dc049-clz92" Oct 30 13:06:39.495807 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:39.495777 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb559\" (UniqueName: \"kubernetes.io/projected/3ca8cd79-0cc2-4813-aa69-e656ab2a1718-kube-api-access-rb559\") pod \"maas-api-revoke-sub-015dc049-clz92\" (UID: \"3ca8cd79-0cc2-4813-aa69-e656ab2a1718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015dc049-clz92" Oct 30 13:06:39.495999 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:39.495979 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ca8cd79-0cc2-4813-aa69-e656ab2a1718-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-015dc049-clz92\" (UID: \"3ca8cd79-0cc2-4813-aa69-e656ab2a1718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015dc049-clz92" Oct 30 13:06:39.496107 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:39.496090 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ca8cd79-0cc2-4813-aa69-e656ab2a1718-maas-api-xks-ca\") pod \"maas-api-revoke-sub-015dc049-clz92\" (UID: \"3ca8cd79-0cc2-4813-aa69-e656ab2a1718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015dc049-clz92" Oct 30 13:06:39.496656 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:39.496633 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ca8cd79-0cc2-4813-aa69-e656ab2a1718-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-015dc049-clz92\" (UID: \"3ca8cd79-0cc2-4813-aa69-e656ab2a1718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015dc049-clz92" Oct 30 13:06:39.498529 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:39.498503 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ca8cd79-0cc2-4813-aa69-e656ab2a1718-maas-api-xks-ca\") pod \"maas-api-revoke-sub-015dc049-clz92\" (UID: \"3ca8cd79-0cc2-4813-aa69-e656ab2a1718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015dc049-clz92" Oct 30 13:06:39.523538 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:39.523510 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb559\" (UniqueName: \"kubernetes.io/projected/3ca8cd79-0cc2-4813-aa69-e656ab2a1718-kube-api-access-rb559\") pod \"maas-api-revoke-sub-015dc049-clz92\" (UID: \"3ca8cd79-0cc2-4813-aa69-e656ab2a1718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015dc049-clz92" Oct 30 13:06:39.623883 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:39.623808 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015dc049-clz92" Oct 30 13:06:39.953802 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:39.953772 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-015dc049-clz92"] Oct 30 13:06:39.954006 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:06:39.953981 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca8cd79_0cc2_4813_aa69_e656ab2a1718.slice/crio-d0fd7daaf7a7ffb877a4fbcc7757ce3aab647369f2bb06da7701eed933969f23 WatchSource:0}: Error finding container d0fd7daaf7a7ffb877a4fbcc7757ce3aab647369f2bb06da7701eed933969f23: Status 404 returned error can't find the container with id d0fd7daaf7a7ffb877a4fbcc7757ce3aab647369f2bb06da7701eed933969f23 Oct 30 13:06:40.545515 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:40.545480 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015dc049-clz92" event={"ID":"3ca8cd79-0cc2-4813-aa69-e656ab2a1718","Type":"ContainerStarted","Data":"8c7856d48f7d37f5f7a2d314389687689585844b4a1bb7cf9d4ee87214b9af16"} Oct 30 13:06:40.545926 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:40.545540 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015dc049-clz92" event={"ID":"3ca8cd79-0cc2-4813-aa69-e656ab2a1718","Type":"ContainerStarted","Data":"d0fd7daaf7a7ffb877a4fbcc7757ce3aab647369f2bb06da7701eed933969f23"} Oct 30 13:06:40.605070 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:40.604968 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015dc049-clz92" podStartSLOduration=1.6049532549999999 podStartE2EDuration="1.604953255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:40.589261083 +0000 UTC m=+1299.192759530" watchObservedRunningTime="2026-10-30 13:06:40.604953255 +0000 UTC m=+1299.208451701" Oct 30 13:06:41.554426 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:41.554395 4931 generic.go:356] "Generic (PLEG): container finished" podID="3ca8cd79-0cc2-4813-aa69-e656ab2a1718" containerID="8c7856d48f7d37f5f7a2d314389687689585844b4a1bb7cf9d4ee87214b9af16" exitCode=0 Oct 30 13:06:41.554819 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:41.554474 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015dc049-clz92" event={"ID":"3ca8cd79-0cc2-4813-aa69-e656ab2a1718","Type":"ContainerDied","Data":"8c7856d48f7d37f5f7a2d314389687689585844b4a1bb7cf9d4ee87214b9af16"} Oct 30 13:06:42.732231 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:42.732204 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015dc049-clz92" Oct 30 13:06:42.784245 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:42.784200 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ca8cd79-0cc2-4813-aa69-e656ab2a1718-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ca8cd79-0cc2-4813-aa69-e656ab2a1718-maas-api-xks-ca\") pod \"3ca8cd79-0cc2-4813-aa69-e656ab2a1718\" (UID: \"3ca8cd79-0cc2-4813-aa69-e656ab2a1718\") " Oct 30 13:06:42.784471 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:42.784354 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ca8cd79-0cc2-4813-aa69-e656ab2a1718-kube-api-access-rb559\" (UniqueName: \"kubernetes.io/projected/3ca8cd79-0cc2-4813-aa69-e656ab2a1718-kube-api-access-rb559\") pod \"3ca8cd79-0cc2-4813-aa69-e656ab2a1718\" (UID: \"3ca8cd79-0cc2-4813-aa69-e656ab2a1718\") " Oct 30 13:06:42.784539 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:42.784472 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ca8cd79-0cc2-4813-aa69-e656ab2a1718-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ca8cd79-0cc2-4813-aa69-e656ab2a1718-maas-api-openshift-ca\") pod \"3ca8cd79-0cc2-4813-aa69-e656ab2a1718\" (UID: \"3ca8cd79-0cc2-4813-aa69-e656ab2a1718\") " Oct 30 13:06:42.785384 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:42.785355 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ca8cd79-0cc2-4813-aa69-e656ab2a1718-maas-api-openshift-ca" pod "3ca8cd79-0cc2-4813-aa69-e656ab2a1718" (UID: "3ca8cd79-0cc2-4813-aa69-e656ab2a1718"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:42.788050 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:42.788020 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ca8cd79-0cc2-4813-aa69-e656ab2a1718-maas-api-xks-ca" pod "3ca8cd79-0cc2-4813-aa69-e656ab2a1718" (UID: "3ca8cd79-0cc2-4813-aa69-e656ab2a1718"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:42.788432 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:42.788404 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ca8cd79-0cc2-4813-aa69-e656ab2a1718-kube-api-access-rb559" pod "3ca8cd79-0cc2-4813-aa69-e656ab2a1718" (UID: "3ca8cd79-0cc2-4813-aa69-e656ab2a1718"). InnerVolumeSpecName "kube-api-access-rb559". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:42.887190 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:42.887066 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ca8cd79-0cc2-4813-aa69-e656ab2a1718-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:06:42.887190 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:42.887112 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb559\" (UniqueName: \"kubernetes.io/projected/3ca8cd79-0cc2-4813-aa69-e656ab2a1718-kube-api-access-rb559\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:06:42.887190 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:42.887126 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ca8cd79-0cc2-4813-aa69-e656ab2a1718-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:06:43.573762 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:43.573727 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015dc049-clz92" Oct 30 13:06:43.573906 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:43.573762 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015dc049-clz92" event={"ID":"3ca8cd79-0cc2-4813-aa69-e656ab2a1718","Type":"ContainerDied","Data":"d0fd7daaf7a7ffb877a4fbcc7757ce3aab647369f2bb06da7701eed933969f23"} Oct 30 13:06:43.573906 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:43.573811 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0fd7daaf7a7ffb877a4fbcc7757ce3aab647369f2bb06da7701eed933969f23" Oct 30 13:06:51.726833 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:51.726794 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5af2de67-xd5xc"] Oct 30 13:06:51.727435 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:51.727420 4931 cpu_manager.go:414] "containerMap: removing container" podUID="3ca8cd79-0cc2-4813-aa69-e656ab2a1718" containerName="revoke-keys" Oct 30 13:06:51.727435 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:51.727436 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ca8cd79-0cc2-4813-aa69-e656ab2a1718" containerName="revoke-keys" Oct 30 13:06:51.727549 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:51.727535 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ca8cd79-0cc2-4813-aa69-e656ab2a1718" containerName="revoke-keys" Oct 30 13:06:51.730984 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:51.730968 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5af2de67-xd5xc" Oct 30 13:06:51.750665 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:51.750635 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5af2de67-xd5xc"] Oct 30 13:06:51.758622 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:51.758597 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:06:51.758752 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:51.758642 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:06:51.828601 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:51.828573 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgxjq\" (UniqueName: \"kubernetes.io/projected/95c91816-d868-422b-9a5d-35f5038f3549-kube-api-access-qgxjq\") pod \"maas-api-revoke-keys-e2e-derive-5af2de67-xd5xc\" (UID: \"95c91816-d868-422b-9a5d-35f5038f3549\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5af2de67-xd5xc" Oct 30 13:06:51.828732 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:51.828708 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95c91816-d868-422b-9a5d-35f5038f3549-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-5af2de67-xd5xc\" (UID: \"95c91816-d868-422b-9a5d-35f5038f3549\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5af2de67-xd5xc" Oct 30 13:06:51.828896 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:51.828873 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95c91816-d868-422b-9a5d-35f5038f3549-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-5af2de67-xd5xc\" (UID: \"95c91816-d868-422b-9a5d-35f5038f3549\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5af2de67-xd5xc" Oct 30 13:06:51.932292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:51.931730 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95c91816-d868-422b-9a5d-35f5038f3549-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-5af2de67-xd5xc\" (UID: \"95c91816-d868-422b-9a5d-35f5038f3549\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5af2de67-xd5xc" Oct 30 13:06:51.932292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:51.932025 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95c91816-d868-422b-9a5d-35f5038f3549-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-5af2de67-xd5xc\" (UID: \"95c91816-d868-422b-9a5d-35f5038f3549\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5af2de67-xd5xc" Oct 30 13:06:51.932522 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:51.932255 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgxjq\" (UniqueName: \"kubernetes.io/projected/95c91816-d868-422b-9a5d-35f5038f3549-kube-api-access-qgxjq\") pod \"maas-api-revoke-keys-e2e-derive-5af2de67-xd5xc\" (UID: \"95c91816-d868-422b-9a5d-35f5038f3549\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5af2de67-xd5xc" Oct 30 13:06:51.935634 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:51.934016 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95c91816-d868-422b-9a5d-35f5038f3549-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-5af2de67-xd5xc\" (UID: \"95c91816-d868-422b-9a5d-35f5038f3549\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5af2de67-xd5xc" Oct 30 13:06:51.935887 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:51.935705 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95c91816-d868-422b-9a5d-35f5038f3549-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-5af2de67-xd5xc\" (UID: \"95c91816-d868-422b-9a5d-35f5038f3549\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5af2de67-xd5xc" Oct 30 13:06:51.957986 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:51.957961 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgxjq\" (UniqueName: \"kubernetes.io/projected/95c91816-d868-422b-9a5d-35f5038f3549-kube-api-access-qgxjq\") pod \"maas-api-revoke-keys-e2e-derive-5af2de67-xd5xc\" (UID: \"95c91816-d868-422b-9a5d-35f5038f3549\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5af2de67-xd5xc" Oct 30 13:06:52.047085 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:52.046994 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5af2de67-xd5xc" Oct 30 13:06:52.359722 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:06:52.359687 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c91816_d868_422b_9a5d_35f5038f3549.slice/crio-9c29b768710139fcd0298ba6c4ac430889bee22c53fcefc51b9fb14cd7c7dd92 WatchSource:0}: Error finding container 9c29b768710139fcd0298ba6c4ac430889bee22c53fcefc51b9fb14cd7c7dd92: Status 404 returned error can't find the container with id 9c29b768710139fcd0298ba6c4ac430889bee22c53fcefc51b9fb14cd7c7dd92 Oct 30 13:06:52.359851 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:52.359811 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5af2de67-xd5xc"] Oct 30 13:06:52.653315 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:52.653289 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5af2de67-xd5xc" event={"ID":"95c91816-d868-422b-9a5d-35f5038f3549","Type":"ContainerStarted","Data":"9c29b768710139fcd0298ba6c4ac430889bee22c53fcefc51b9fb14cd7c7dd92"} Oct 30 13:06:53.662341 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:53.662301 4931 generic.go:356] "Generic (PLEG): container finished" podID="95c91816-d868-422b-9a5d-35f5038f3549" containerID="f9ef3e48b4ebebd3ceeadf20cc81b07e460763bc761c480752c048aaf332e28a" exitCode=0 Oct 30 13:06:53.662710 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:53.662359 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5af2de67-xd5xc" event={"ID":"95c91816-d868-422b-9a5d-35f5038f3549","Type":"ContainerDied","Data":"f9ef3e48b4ebebd3ceeadf20cc81b07e460763bc761c480752c048aaf332e28a"} Oct 30 13:06:54.851142 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:54.851117 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5af2de67-xd5xc" Oct 30 13:06:54.984551 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:54.984522 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95c91816-d868-422b-9a5d-35f5038f3549-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95c91816-d868-422b-9a5d-35f5038f3549-maas-api-xks-ca\") pod \"95c91816-d868-422b-9a5d-35f5038f3549\" (UID: \"95c91816-d868-422b-9a5d-35f5038f3549\") " Oct 30 13:06:54.984717 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:54.984691 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95c91816-d868-422b-9a5d-35f5038f3549-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95c91816-d868-422b-9a5d-35f5038f3549-maas-api-openshift-ca\") pod \"95c91816-d868-422b-9a5d-35f5038f3549\" (UID: \"95c91816-d868-422b-9a5d-35f5038f3549\") " Oct 30 13:06:54.984760 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:54.984719 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95c91816-d868-422b-9a5d-35f5038f3549-kube-api-access-qgxjq\" (UniqueName: \"kubernetes.io/projected/95c91816-d868-422b-9a5d-35f5038f3549-kube-api-access-qgxjq\") pod \"95c91816-d868-422b-9a5d-35f5038f3549\" (UID: \"95c91816-d868-422b-9a5d-35f5038f3549\") " Oct 30 13:06:54.985018 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:54.984984 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95c91816-d868-422b-9a5d-35f5038f3549-maas-api-openshift-ca" pod "95c91816-d868-422b-9a5d-35f5038f3549" (UID: "95c91816-d868-422b-9a5d-35f5038f3549"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:54.985276 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:54.985244 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95c91816-d868-422b-9a5d-35f5038f3549-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:06:54.986657 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:54.986624 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95c91816-d868-422b-9a5d-35f5038f3549-maas-api-xks-ca" pod "95c91816-d868-422b-9a5d-35f5038f3549" (UID: "95c91816-d868-422b-9a5d-35f5038f3549"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:54.986751 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:54.986703 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c91816-d868-422b-9a5d-35f5038f3549-kube-api-access-qgxjq" pod "95c91816-d868-422b-9a5d-35f5038f3549" (UID: "95c91816-d868-422b-9a5d-35f5038f3549"). InnerVolumeSpecName "kube-api-access-qgxjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:55.086693 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:55.086663 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95c91816-d868-422b-9a5d-35f5038f3549-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:06:55.086693 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:55.086689 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgxjq\" (UniqueName: \"kubernetes.io/projected/95c91816-d868-422b-9a5d-35f5038f3549-kube-api-access-qgxjq\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:06:55.678019 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:55.677988 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5af2de67-xd5xc" Oct 30 13:06:55.678208 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:55.678025 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5af2de67-xd5xc" event={"ID":"95c91816-d868-422b-9a5d-35f5038f3549","Type":"ContainerDied","Data":"9c29b768710139fcd0298ba6c4ac430889bee22c53fcefc51b9fb14cd7c7dd92"} Oct 30 13:06:55.678208 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:06:55.678073 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c29b768710139fcd0298ba6c4ac430889bee22c53fcefc51b9fb14cd7c7dd92" Oct 30 13:07:00.969023 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:00.968988 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:07:01.745201 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:01.745169 4931 scope.go:122] "RemoveContainer" containerID="c4e976747dc6fd8ee8ba3a1bf81137e440610c84df416b09b66618b109fb0d04" Oct 30 13:07:01.756137 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:01.756114 4931 scope.go:122] "RemoveContainer" containerID="7648a74916883ae5d2e60c855a47e4066871dd0291f88ae2c1490c31afc4bba7" Oct 30 13:07:01.767642 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:01.767616 4931 scope.go:122] "RemoveContainer" containerID="457caf1d787225da2e7025663c1ee42ce8b539ae9483a01f258f16a5c507b8f0" Oct 30 13:07:03.599784 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:03.599741 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-ddspx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:07:03.600397 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:03.600360 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx" podUID="d126cf3a-50cd-45e4-b37f-b268fdbb3d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:04.308482 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:04.308441 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-tvm7w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:07:04.308701 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:04.308564 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" podUID="3dd90613-12ac-4509-9f2e-b5c1e199719f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:07.076072 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:07.076024 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5af2de67-xd5xc"] Oct 30 13:07:07.087770 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:07.087732 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5af2de67-xd5xc"] Oct 30 13:07:07.355542 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:07:07.355450 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:07:07.516876 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:07.516843 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95c91816-d868-422b-9a5d-35f5038f3549" path="/var/lib/kubelet/pods/95c91816-d868-422b-9a5d-35f5038f3549/volumes" Oct 30 13:07:13.046463 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:13.046428 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80c0ff30-jmnwl"] Oct 30 13:07:13.058949 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:13.058920 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80c0ff30-jmnwl"] Oct 30 13:07:13.514884 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:13.514848 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b32e46d-cec9-4453-aebc-f4f45301146c" path="/var/lib/kubelet/pods/1b32e46d-cec9-4453-aebc-f4f45301146c/volumes" Oct 30 13:07:32.109873 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:32.109826 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be4bb9-hl88w"] Oct 30 13:07:32.110530 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:32.110515 4931 cpu_manager.go:414] "containerMap: removing container" podUID="95c91816-d868-422b-9a5d-35f5038f3549" containerName="revoke-keys" Oct 30 13:07:32.110588 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:32.110532 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95c91816-d868-422b-9a5d-35f5038f3549" containerName="revoke-keys" Oct 30 13:07:32.110648 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:32.110638 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="95c91816-d868-422b-9a5d-35f5038f3549" containerName="revoke-keys" Oct 30 13:07:32.114459 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:32.114441 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be4bb9-hl88w" Oct 30 13:07:32.133774 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:32.133744 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be4bb9-hl88w"] Oct 30 13:07:32.142586 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:32.142557 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:07:32.142728 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:32.142593 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:07:32.275465 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:32.275428 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sth2z\" (UniqueName: \"kubernetes.io/projected/a4e83be0-d82f-4b17-86ab-039c5250fba4-kube-api-access-sth2z\") pod \"maas-api-revoke-keys-e2e-migrate-85be4bb9-hl88w\" (UID: \"a4e83be0-d82f-4b17-86ab-039c5250fba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be4bb9-hl88w" Oct 30 13:07:32.275694 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:32.275595 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4e83be0-d82f-4b17-86ab-039c5250fba4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-85be4bb9-hl88w\" (UID: \"a4e83be0-d82f-4b17-86ab-039c5250fba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be4bb9-hl88w" Oct 30 13:07:32.275694 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:32.275652 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4e83be0-d82f-4b17-86ab-039c5250fba4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-85be4bb9-hl88w\" (UID: \"a4e83be0-d82f-4b17-86ab-039c5250fba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be4bb9-hl88w" Oct 30 13:07:32.376664 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:32.376580 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sth2z\" (UniqueName: \"kubernetes.io/projected/a4e83be0-d82f-4b17-86ab-039c5250fba4-kube-api-access-sth2z\") pod \"maas-api-revoke-keys-e2e-migrate-85be4bb9-hl88w\" (UID: \"a4e83be0-d82f-4b17-86ab-039c5250fba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be4bb9-hl88w" Oct 30 13:07:32.376834 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:32.376669 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4e83be0-d82f-4b17-86ab-039c5250fba4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-85be4bb9-hl88w\" (UID: \"a4e83be0-d82f-4b17-86ab-039c5250fba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be4bb9-hl88w" Oct 30 13:07:32.376834 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:32.376709 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4e83be0-d82f-4b17-86ab-039c5250fba4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-85be4bb9-hl88w\" (UID: \"a4e83be0-d82f-4b17-86ab-039c5250fba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be4bb9-hl88w" Oct 30 13:07:32.377429 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:32.377407 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4e83be0-d82f-4b17-86ab-039c5250fba4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-85be4bb9-hl88w\" (UID: \"a4e83be0-d82f-4b17-86ab-039c5250fba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be4bb9-hl88w" Oct 30 13:07:32.379006 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:32.378984 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4e83be0-d82f-4b17-86ab-039c5250fba4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-85be4bb9-hl88w\" (UID: \"a4e83be0-d82f-4b17-86ab-039c5250fba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be4bb9-hl88w" Oct 30 13:07:32.407791 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:32.407763 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sth2z\" (UniqueName: \"kubernetes.io/projected/a4e83be0-d82f-4b17-86ab-039c5250fba4-kube-api-access-sth2z\") pod \"maas-api-revoke-keys-e2e-migrate-85be4bb9-hl88w\" (UID: \"a4e83be0-d82f-4b17-86ab-039c5250fba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be4bb9-hl88w" Oct 30 13:07:32.428603 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:32.428581 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be4bb9-hl88w" Oct 30 13:07:32.737485 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:07:32.737453 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4e83be0_d82f_4b17_86ab_039c5250fba4.slice/crio-3bb7174161c83694881f4738b42c54f1e1cc05688f51251c9cdc4611f2114fc3 WatchSource:0}: Error finding container 3bb7174161c83694881f4738b42c54f1e1cc05688f51251c9cdc4611f2114fc3: Status 404 returned error can't find the container with id 3bb7174161c83694881f4738b42c54f1e1cc05688f51251c9cdc4611f2114fc3 Oct 30 13:07:32.738568 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:32.738544 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be4bb9-hl88w"] Oct 30 13:07:33.096284 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:33.096232 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be4bb9-hl88w" event={"ID":"a4e83be0-d82f-4b17-86ab-039c5250fba4","Type":"ContainerStarted","Data":"9e6003ba488b867845bc7dc42d7e6ca45940185432dc3c2654ab77d894d42625"} Oct 30 13:07:33.096469 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:33.096331 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be4bb9-hl88w" event={"ID":"a4e83be0-d82f-4b17-86ab-039c5250fba4","Type":"ContainerStarted","Data":"3bb7174161c83694881f4738b42c54f1e1cc05688f51251c9cdc4611f2114fc3"} Oct 30 13:07:33.145729 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:33.145678 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be4bb9-hl88w" podStartSLOduration=1.145662588 podStartE2EDuration="1.145662588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:33.141868177 +0000 UTC m=+1351.745366622" watchObservedRunningTime="2026-10-30 13:07:33.145662588 +0000 UTC m=+1351.749161034" Oct 30 13:07:34.105784 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:34.105749 4931 generic.go:356] "Generic (PLEG): container finished" podID="a4e83be0-d82f-4b17-86ab-039c5250fba4" containerID="9e6003ba488b867845bc7dc42d7e6ca45940185432dc3c2654ab77d894d42625" exitCode=0 Oct 30 13:07:34.105962 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:34.105832 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be4bb9-hl88w" event={"ID":"a4e83be0-d82f-4b17-86ab-039c5250fba4","Type":"ContainerDied","Data":"9e6003ba488b867845bc7dc42d7e6ca45940185432dc3c2654ab77d894d42625"} Oct 30 13:07:35.311411 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:35.311379 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be4bb9-hl88w" Oct 30 13:07:35.420659 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:35.420566 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4e83be0-d82f-4b17-86ab-039c5250fba4-kube-api-access-sth2z\" (UniqueName: \"kubernetes.io/projected/a4e83be0-d82f-4b17-86ab-039c5250fba4-kube-api-access-sth2z\") pod \"a4e83be0-d82f-4b17-86ab-039c5250fba4\" (UID: \"a4e83be0-d82f-4b17-86ab-039c5250fba4\") " Oct 30 13:07:35.420659 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:35.420606 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4e83be0-d82f-4b17-86ab-039c5250fba4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4e83be0-d82f-4b17-86ab-039c5250fba4-maas-api-openshift-ca\") pod \"a4e83be0-d82f-4b17-86ab-039c5250fba4\" (UID: \"a4e83be0-d82f-4b17-86ab-039c5250fba4\") " Oct 30 13:07:35.420893 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:35.420709 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4e83be0-d82f-4b17-86ab-039c5250fba4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4e83be0-d82f-4b17-86ab-039c5250fba4-maas-api-xks-ca\") pod \"a4e83be0-d82f-4b17-86ab-039c5250fba4\" (UID: \"a4e83be0-d82f-4b17-86ab-039c5250fba4\") " Oct 30 13:07:35.421021 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:35.420995 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4e83be0-d82f-4b17-86ab-039c5250fba4-maas-api-openshift-ca" pod "a4e83be0-d82f-4b17-86ab-039c5250fba4" (UID: "a4e83be0-d82f-4b17-86ab-039c5250fba4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:35.421436 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:35.421398 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4e83be0-d82f-4b17-86ab-039c5250fba4-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:07:35.423092 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:35.423064 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4e83be0-d82f-4b17-86ab-039c5250fba4-kube-api-access-sth2z" pod "a4e83be0-d82f-4b17-86ab-039c5250fba4" (UID: "a4e83be0-d82f-4b17-86ab-039c5250fba4"). InnerVolumeSpecName "kube-api-access-sth2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:35.423212 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:35.423189 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4e83be0-d82f-4b17-86ab-039c5250fba4-maas-api-xks-ca" pod "a4e83be0-d82f-4b17-86ab-039c5250fba4" (UID: "a4e83be0-d82f-4b17-86ab-039c5250fba4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:35.522815 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:35.522758 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sth2z\" (UniqueName: \"kubernetes.io/projected/a4e83be0-d82f-4b17-86ab-039c5250fba4-kube-api-access-sth2z\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:07:35.522815 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:35.522788 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4e83be0-d82f-4b17-86ab-039c5250fba4-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:07:36.125744 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:36.125711 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be4bb9-hl88w" Oct 30 13:07:36.125744 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:36.125741 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be4bb9-hl88w" event={"ID":"a4e83be0-d82f-4b17-86ab-039c5250fba4","Type":"ContainerDied","Data":"3bb7174161c83694881f4738b42c54f1e1cc05688f51251c9cdc4611f2114fc3"} Oct 30 13:07:36.125974 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:36.125793 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bb7174161c83694881f4738b42c54f1e1cc05688f51251c9cdc4611f2114fc3" Oct 30 13:07:37.354360 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:07:37.354322 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:07:39.814765 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:39.813173 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:07:41.781197 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:41.781158 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n"] Oct 30 13:07:41.783664 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:41.782522 4931 cpu_manager.go:414] "containerMap: removing container" podUID="a4e83be0-d82f-4b17-86ab-039c5250fba4" containerName="revoke-keys" Oct 30 13:07:41.783871 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:41.783848 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4e83be0-d82f-4b17-86ab-039c5250fba4" containerName="revoke-keys" Oct 30 13:07:41.784207 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:41.784190 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4e83be0-d82f-4b17-86ab-039c5250fba4" containerName="revoke-keys" Oct 30 13:07:41.789868 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:41.789838 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n" Oct 30 13:07:41.813803 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:41.813776 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-2afe58-serving-cert\"" Oct 30 13:07:41.821452 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:41.821421 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n"] Oct 30 13:07:41.898518 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:41.898480 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1342efa-24d9-414d-ac5b-c4efc78ce33c-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n\" (UID: \"f1342efa-24d9-414d-ac5b-c4efc78ce33c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n" Oct 30 13:07:41.898743 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:41.898706 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn8m7\" (UniqueName: \"kubernetes.io/projected/f1342efa-24d9-414d-ac5b-c4efc78ce33c-kube-api-access-rn8m7\") pod \"maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n\" (UID: \"f1342efa-24d9-414d-ac5b-c4efc78ce33c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n" Oct 30 13:07:41.898869 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:41.898851 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1342efa-24d9-414d-ac5b-c4efc78ce33c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n\" (UID: \"f1342efa-24d9-414d-ac5b-c4efc78ce33c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n" Oct 30 13:07:41.898986 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:41.898969 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1342efa-24d9-414d-ac5b-c4efc78ce33c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n\" (UID: \"f1342efa-24d9-414d-ac5b-c4efc78ce33c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n" Oct 30 13:07:42.000087 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:42.000053 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1342efa-24d9-414d-ac5b-c4efc78ce33c-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n\" (UID: \"f1342efa-24d9-414d-ac5b-c4efc78ce33c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n" Oct 30 13:07:42.000298 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:42.000201 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn8m7\" (UniqueName: \"kubernetes.io/projected/f1342efa-24d9-414d-ac5b-c4efc78ce33c-kube-api-access-rn8m7\") pod \"maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n\" (UID: \"f1342efa-24d9-414d-ac5b-c4efc78ce33c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n" Oct 30 13:07:42.000388 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:42.000368 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1342efa-24d9-414d-ac5b-c4efc78ce33c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n\" (UID: \"f1342efa-24d9-414d-ac5b-c4efc78ce33c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n" Oct 30 13:07:42.000475 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:42.000461 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1342efa-24d9-414d-ac5b-c4efc78ce33c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n\" (UID: \"f1342efa-24d9-414d-ac5b-c4efc78ce33c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n" Oct 30 13:07:42.000642 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:07:42.000516 4931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2afe58-serving-cert: secret "maas-api-e2e-shared-a-w1-2afe58-serving-cert" not found Oct 30 13:07:42.000785 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:07:42.000701 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1342efa-24d9-414d-ac5b-c4efc78ce33c-maas-api-tls podName:f1342efa-24d9-414d-ac5b-c4efc78ce33c nodeName:}" failed. No retries permitted until 2026-10-30 13:07:42.500677353 +0000 UTC m=+1361.104175821 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f1342efa-24d9-414d-ac5b-c4efc78ce33c-maas-api-tls") pod "maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n" (UID: "f1342efa-24d9-414d-ac5b-c4efc78ce33c") : secret "maas-api-e2e-shared-a-w1-2afe58-serving-cert" not found Oct 30 13:07:42.001106 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:42.001080 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1342efa-24d9-414d-ac5b-c4efc78ce33c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n\" (UID: \"f1342efa-24d9-414d-ac5b-c4efc78ce33c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n" Oct 30 13:07:42.002651 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:42.002628 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1342efa-24d9-414d-ac5b-c4efc78ce33c-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n\" (UID: \"f1342efa-24d9-414d-ac5b-c4efc78ce33c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n" Oct 30 13:07:42.026695 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:42.026613 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn8m7\" (UniqueName: \"kubernetes.io/projected/f1342efa-24d9-414d-ac5b-c4efc78ce33c-kube-api-access-rn8m7\") pod \"maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n\" (UID: \"f1342efa-24d9-414d-ac5b-c4efc78ce33c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n" Oct 30 13:07:42.507628 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:42.507598 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1342efa-24d9-414d-ac5b-c4efc78ce33c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n\" (UID: \"f1342efa-24d9-414d-ac5b-c4efc78ce33c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n" Oct 30 13:07:42.510037 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:42.510006 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1342efa-24d9-414d-ac5b-c4efc78ce33c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n\" (UID: \"f1342efa-24d9-414d-ac5b-c4efc78ce33c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n" Oct 30 13:07:42.714066 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:42.714033 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n" Oct 30 13:07:43.113311 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:07:43.113280 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1342efa_24d9_414d_ac5b_c4efc78ce33c.slice/crio-6a2e17819d52a3afc74a5fd341010e123bbc2412debd481fc43e17c494960bcb WatchSource:0}: Error finding container 6a2e17819d52a3afc74a5fd341010e123bbc2412debd481fc43e17c494960bcb: Status 404 returned error can't find the container with id 6a2e17819d52a3afc74a5fd341010e123bbc2412debd481fc43e17c494960bcb Oct 30 13:07:43.115193 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:43.115167 4931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:07:43.115327 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:43.115190 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n"] Oct 30 13:07:43.183692 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:43.183659 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n" event={"ID":"f1342efa-24d9-414d-ac5b-c4efc78ce33c","Type":"ContainerStarted","Data":"6a2e17819d52a3afc74a5fd341010e123bbc2412debd481fc43e17c494960bcb"} Oct 30 13:07:44.192439 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:44.192401 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n" event={"ID":"f1342efa-24d9-414d-ac5b-c4efc78ce33c","Type":"ContainerStarted","Data":"fac285f6638d54dc8f1c242f97d82d6d977fee4a8fb551aa0619b245ee3291c6"} Oct 30 13:07:44.192975 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:44.192498 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n" Oct 30 13:07:44.240908 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:44.240857 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n" podStartSLOduration=2.745519539 podStartE2EDuration="3.240843563s" totalImagesPullingTime="495.324024ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:44.237379553 +0000 UTC m=+1362.840878019" watchObservedRunningTime="2026-10-30 13:07:44.240843563 +0000 UTC m=+1362.844342008" Oct 30 13:07:46.990766 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:46.990718 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be4bb9-hl88w"] Oct 30 13:07:47.004300 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:47.004261 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be4bb9-hl88w"] Oct 30 13:07:47.561184 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:47.561150 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4e83be0-d82f-4b17-86ab-039c5250fba4" path="/var/lib/kubelet/pods/a4e83be0-d82f-4b17-86ab-039c5250fba4/volumes" Oct 30 13:07:50.206214 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:07:50.206181 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n" Oct 30 13:08:01.805819 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:01.805787 4931 scope.go:122] "RemoveContainer" containerID="b00a6d043a35b67fb1520621cc56992f027891345508cb630d2d8a2b8ff16368" Oct 30 13:08:01.826508 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:01.826486 4931 scope.go:122] "RemoveContainer" containerID="5d913bfbf01a84da37eda446c8a3acc2e61e3e95f4029a986a6dd04ff98cc52f" Oct 30 13:08:07.353628 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:08:07.353588 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:08:09.041479 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:09.041429 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc4896a9-25mnm"] Oct 30 13:08:09.059678 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:09.059645 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc4896a9-25mnm"] Oct 30 13:08:09.520711 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:09.520671 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59fe0cc8-1d0a-4b68-9111-070ffa1d4a08" path="/var/lib/kubelet/pods/59fe0cc8-1d0a-4b68-9111-070ffa1d4a08/volumes" Oct 30 13:08:17.776171 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:17.776084 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:08:31.057236 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:31.057197 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-946d7aa1-v7flq"] Oct 30 13:08:31.070635 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:31.070601 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-946d7aa1-v7flq"] Oct 30 13:08:31.515896 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:31.515860 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58b1995b-761f-4aad-9de2-2c4910eed9f2" path="/var/lib/kubelet/pods/58b1995b-761f-4aad-9de2-2c4910eed9f2/volumes" Oct 30 13:08:33.230760 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:33.230729 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs"] Oct 30 13:08:33.234706 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:33.234684 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" Oct 30 13:08:33.268016 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:33.267983 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-1c0e6c42\"/\"kube-root-ca.crt\"" Oct 30 13:08:33.268247 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:33.268229 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-1c0e6c42\"/\"default-dockercfg-8qp7q\"" Oct 30 13:08:33.268487 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:33.268465 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-1c0e6c42\"/\"e2e-lifecycle-model-1c0e6c42-kserve-self-signed-certs\"" Oct 30 13:08:33.274527 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:33.274492 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs"] Oct 30 13:08:33.354076 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:33.354045 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-1c0e6c42\"/\"openshift-service-ca.crt\"" Oct 30 13:08:33.364999 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:33.364970 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-tmp-dir\") pod \"e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs\" (UID: \"fb4d115a-ce18-42fe-a26e-d28f0cbadd1e\") " pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" Oct 30 13:08:33.365176 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:33.365041 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-model-cache\") pod \"e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs\" (UID: \"fb4d115a-ce18-42fe-a26e-d28f0cbadd1e\") " pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" Oct 30 13:08:33.365325 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:33.365299 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-tls-certs\") pod \"e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs\" (UID: \"fb4d115a-ce18-42fe-a26e-d28f0cbadd1e\") " pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" Oct 30 13:08:33.365380 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:33.365365 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-home\") pod \"e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs\" (UID: \"fb4d115a-ce18-42fe-a26e-d28f0cbadd1e\") " pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" Oct 30 13:08:33.365517 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:33.365499 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnmnx\" (UniqueName: \"kubernetes.io/projected/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-kube-api-access-pnmnx\") pod \"e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs\" (UID: \"fb4d115a-ce18-42fe-a26e-d28f0cbadd1e\") " pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" Oct 30 13:08:33.365562 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:33.365548 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-dshm\") pod \"e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs\" (UID: \"fb4d115a-ce18-42fe-a26e-d28f0cbadd1e\") " pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" Oct 30 13:08:33.467608 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:33.467568 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-model-cache\") pod \"e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs\" (UID: \"fb4d115a-ce18-42fe-a26e-d28f0cbadd1e\") " pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" Oct 30 13:08:33.467827 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:33.467744 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-tls-certs\") pod \"e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs\" (UID: \"fb4d115a-ce18-42fe-a26e-d28f0cbadd1e\") " pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" Oct 30 13:08:33.467827 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:33.467803 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-home\") pod \"e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs\" (UID: \"fb4d115a-ce18-42fe-a26e-d28f0cbadd1e\") " pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" Oct 30 13:08:33.467957 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:33.467901 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnmnx\" (UniqueName: \"kubernetes.io/projected/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-kube-api-access-pnmnx\") pod \"e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs\" (UID: \"fb4d115a-ce18-42fe-a26e-d28f0cbadd1e\") " pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" Oct 30 13:08:33.467957 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:33.467943 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-dshm\") pod \"e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs\" (UID: \"fb4d115a-ce18-42fe-a26e-d28f0cbadd1e\") " pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" Oct 30 13:08:33.468079 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:33.468053 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-model-cache\") pod \"e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs\" (UID: \"fb4d115a-ce18-42fe-a26e-d28f0cbadd1e\") " pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" Oct 30 13:08:33.468136 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:33.468058 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-tmp-dir\") pod \"e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs\" (UID: \"fb4d115a-ce18-42fe-a26e-d28f0cbadd1e\") " pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" Oct 30 13:08:33.468136 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:33.468098 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-home\") pod \"e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs\" (UID: \"fb4d115a-ce18-42fe-a26e-d28f0cbadd1e\") " pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" Oct 30 13:08:33.468417 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:33.468362 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-tmp-dir\") pod \"e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs\" (UID: \"fb4d115a-ce18-42fe-a26e-d28f0cbadd1e\") " pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" Oct 30 13:08:33.470231 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:33.470205 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-dshm\") pod \"e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs\" (UID: \"fb4d115a-ce18-42fe-a26e-d28f0cbadd1e\") " pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" Oct 30 13:08:33.470509 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:33.470488 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-tls-certs\") pod \"e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs\" (UID: \"fb4d115a-ce18-42fe-a26e-d28f0cbadd1e\") " pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" Oct 30 13:08:33.545096 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:33.544976 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnmnx\" (UniqueName: \"kubernetes.io/projected/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-kube-api-access-pnmnx\") pod \"e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs\" (UID: \"fb4d115a-ce18-42fe-a26e-d28f0cbadd1e\") " pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" Oct 30 13:08:33.550649 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:33.550613 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" Oct 30 13:08:33.992300 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:33.992253 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs"] Oct 30 13:08:33.993330 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:08:33.993301 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb4d115a_ce18_42fe_a26e_d28f0cbadd1e.slice/crio-a3612114dec0a23706fb7f8b93df450c468260f759c75b7d0a33eedf1e959bea WatchSource:0}: Error finding container a3612114dec0a23706fb7f8b93df450c468260f759c75b7d0a33eedf1e959bea: Status 404 returned error can't find the container with id a3612114dec0a23706fb7f8b93df450c468260f759c75b7d0a33eedf1e959bea Oct 30 13:08:34.796322 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:34.796256 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" event={"ID":"fb4d115a-ce18-42fe-a26e-d28f0cbadd1e","Type":"ContainerStarted","Data":"4aaa5fc829c5b89ad97986542dbf5df1bbb4deee1a8a76270c5fe1d6c47c49ed"} Oct 30 13:08:34.796822 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:34.796341 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" event={"ID":"fb4d115a-ce18-42fe-a26e-d28f0cbadd1e","Type":"ContainerStarted","Data":"a3612114dec0a23706fb7f8b93df450c468260f759c75b7d0a33eedf1e959bea"} Oct 30 13:08:35.551600 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:35.551569 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" Oct 30 13:08:37.029892 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:37.029838 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" podStartSLOduration=4.029818245 podStartE2EDuration="4.029818245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:34.84943826 +0000 UTC m=+1413.452936705" watchObservedRunningTime="2026-10-30 13:08:37.029818245 +0000 UTC m=+1415.633316691" Oct 30 13:08:37.032659 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:37.032630 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d"] Oct 30 13:08:37.037100 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:37.037078 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d" Oct 30 13:08:37.050933 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:37.050893 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgxb2\" (UniqueName: \"kubernetes.io/projected/80f8171d-05ef-44b2-af37-12ea086b0334-kube-api-access-qgxb2\") pod \"maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d\" (UID: \"80f8171d-05ef-44b2-af37-12ea086b0334\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d" Oct 30 13:08:37.051087 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:37.051011 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80f8171d-05ef-44b2-af37-12ea086b0334-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d\" (UID: \"80f8171d-05ef-44b2-af37-12ea086b0334\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d" Oct 30 13:08:37.052645 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:37.051284 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80f8171d-05ef-44b2-af37-12ea086b0334-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d\" (UID: \"80f8171d-05ef-44b2-af37-12ea086b0334\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d" Oct 30 13:08:37.052645 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:37.051329 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f8171d-05ef-44b2-af37-12ea086b0334-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d\" (UID: \"80f8171d-05ef-44b2-af37-12ea086b0334\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d" Oct 30 13:08:37.091353 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:37.091315 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-7a71ca-serving-cert\"" Oct 30 13:08:37.107779 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:37.107650 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d"] Oct 30 13:08:37.157190 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:37.153432 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80f8171d-05ef-44b2-af37-12ea086b0334-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d\" (UID: \"80f8171d-05ef-44b2-af37-12ea086b0334\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d" Oct 30 13:08:37.157190 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:37.153733 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80f8171d-05ef-44b2-af37-12ea086b0334-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d\" (UID: \"80f8171d-05ef-44b2-af37-12ea086b0334\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d" Oct 30 13:08:37.157190 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:37.153783 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f8171d-05ef-44b2-af37-12ea086b0334-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d\" (UID: \"80f8171d-05ef-44b2-af37-12ea086b0334\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d" Oct 30 13:08:37.157190 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:37.153827 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgxb2\" (UniqueName: \"kubernetes.io/projected/80f8171d-05ef-44b2-af37-12ea086b0334-kube-api-access-qgxb2\") pod \"maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d\" (UID: \"80f8171d-05ef-44b2-af37-12ea086b0334\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d" Oct 30 13:08:37.157190 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:08:37.153949 4931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a71ca-serving-cert: secret "maas-api-e2e-shared-b-w1-7a71ca-serving-cert" not found Oct 30 13:08:37.157190 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:08:37.154056 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80f8171d-05ef-44b2-af37-12ea086b0334-maas-api-tls podName:80f8171d-05ef-44b2-af37-12ea086b0334 nodeName:}" failed. No retries permitted until 2026-10-30 13:08:37.654035723 +0000 UTC m=+1416.257534156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/80f8171d-05ef-44b2-af37-12ea086b0334-maas-api-tls") pod "maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d" (UID: "80f8171d-05ef-44b2-af37-12ea086b0334") : secret "maas-api-e2e-shared-b-w1-7a71ca-serving-cert" not found Oct 30 13:08:37.157190 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:37.154649 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f8171d-05ef-44b2-af37-12ea086b0334-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d\" (UID: \"80f8171d-05ef-44b2-af37-12ea086b0334\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d" Oct 30 13:08:37.163295 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:37.158549 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80f8171d-05ef-44b2-af37-12ea086b0334-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d\" (UID: \"80f8171d-05ef-44b2-af37-12ea086b0334\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d" Oct 30 13:08:37.184073 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:37.184035 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgxb2\" (UniqueName: \"kubernetes.io/projected/80f8171d-05ef-44b2-af37-12ea086b0334-kube-api-access-qgxb2\") pod \"maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d\" (UID: \"80f8171d-05ef-44b2-af37-12ea086b0334\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d" Oct 30 13:08:37.357258 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:08:37.357169 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:08:37.660407 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:37.660309 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80f8171d-05ef-44b2-af37-12ea086b0334-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d\" (UID: \"80f8171d-05ef-44b2-af37-12ea086b0334\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d" Oct 30 13:08:37.660587 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:08:37.660414 4931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a71ca-serving-cert: secret "maas-api-e2e-shared-b-w1-7a71ca-serving-cert" not found Oct 30 13:08:37.660587 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:08:37.660508 4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80f8171d-05ef-44b2-af37-12ea086b0334-maas-api-tls podName:80f8171d-05ef-44b2-af37-12ea086b0334 nodeName:}" failed. No retries permitted until 2026-10-30 13:08:38.660491469 +0000 UTC m=+1417.263989894 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/80f8171d-05ef-44b2-af37-12ea086b0334-maas-api-tls") pod "maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d" (UID: "80f8171d-05ef-44b2-af37-12ea086b0334") : secret "maas-api-e2e-shared-b-w1-7a71ca-serving-cert" not found Oct 30 13:08:38.678163 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:38.678124 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80f8171d-05ef-44b2-af37-12ea086b0334-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d\" (UID: \"80f8171d-05ef-44b2-af37-12ea086b0334\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d" Oct 30 13:08:38.680644 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:38.680615 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80f8171d-05ef-44b2-af37-12ea086b0334-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d\" (UID: \"80f8171d-05ef-44b2-af37-12ea086b0334\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d" Oct 30 13:08:38.860945 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:38.860915 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d" Oct 30 13:08:39.176114 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:39.176080 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d"] Oct 30 13:08:39.176261 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:08:39.176125 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80f8171d_05ef_44b2_af37_12ea086b0334.slice/crio-cf40229e67d496020e19cf67d139fe60822dd4c0e04f7ebf2a3e8b3c0559a67a WatchSource:0}: Error finding container cf40229e67d496020e19cf67d139fe60822dd4c0e04f7ebf2a3e8b3c0559a67a: Status 404 returned error can't find the container with id cf40229e67d496020e19cf67d139fe60822dd4c0e04f7ebf2a3e8b3c0559a67a Oct 30 13:08:39.838095 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:39.837989 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d" event={"ID":"80f8171d-05ef-44b2-af37-12ea086b0334","Type":"ContainerStarted","Data":"797ebd0f0981f347f9c69624b9368825622cbcb2e41c817ffe3d5264c5b0f228"} Oct 30 13:08:39.838095 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:39.838054 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d" event={"ID":"80f8171d-05ef-44b2-af37-12ea086b0334","Type":"ContainerStarted","Data":"cf40229e67d496020e19cf67d139fe60822dd4c0e04f7ebf2a3e8b3c0559a67a"} Oct 30 13:08:39.838095 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:39.838079 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d" Oct 30 13:08:39.885758 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:39.885695 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d" podStartSLOduration=2.515380233 podStartE2EDuration="2.885677901s" totalImagesPullingTime="370.297668ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:39.883178958 +0000 UTC m=+1418.486677424" watchObservedRunningTime="2026-10-30 13:08:39.885677901 +0000 UTC m=+1418.489176347" Oct 30 13:08:42.032920 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:42.032883 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cdf5ad1-l28dq"] Oct 30 13:08:42.046319 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:42.046286 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cdf5ad1-l28dq"] Oct 30 13:08:43.514551 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:43.514507 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5e5d14e-a36e-46c3-b550-adf7f3a9ad64" path="/var/lib/kubelet/pods/b5e5d14e-a36e-46c3-b550-adf7f3a9ad64/volumes" Oct 30 13:08:43.551935 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:43.551894 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" Oct 30 13:08:43.564402 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:43.564376 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" Oct 30 13:08:43.880838 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:43.880743 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" Oct 30 13:08:45.853590 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:45.853559 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d" Oct 30 13:08:47.336771 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:47.336730 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9"] Oct 30 13:08:47.342238 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:47.342220 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9" Oct 30 13:08:47.373812 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:47.373782 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9"] Oct 30 13:08:47.416007 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:47.415977 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26kf9\" (UID: \"d03b10bb-ca2a-4636-ad2b-69972fbb56cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9" Oct 30 13:08:47.416146 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:47.416037 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26kf9\" (UID: \"d03b10bb-ca2a-4636-ad2b-69972fbb56cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9" Oct 30 13:08:47.416344 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:47.416317 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26kf9\" (UID: \"d03b10bb-ca2a-4636-ad2b-69972fbb56cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9" Oct 30 13:08:47.416415 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:47.416397 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26kf9\" (UID: \"d03b10bb-ca2a-4636-ad2b-69972fbb56cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9" Oct 30 13:08:47.416499 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:47.416481 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26kf9\" (UID: \"d03b10bb-ca2a-4636-ad2b-69972fbb56cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9" Oct 30 13:08:47.416556 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:47.416513 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26kf9\" (UID: \"d03b10bb-ca2a-4636-ad2b-69972fbb56cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9" Oct 30 13:08:47.416613 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:47.416590 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnzdj\" (UniqueName: \"kubernetes.io/projected/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-kube-api-access-mnzdj\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26kf9\" (UID: \"d03b10bb-ca2a-4636-ad2b-69972fbb56cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9" Oct 30 13:08:47.424204 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:47.424185 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 13:08:47.424426 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:47.424412 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 13:08:47.424487 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:47.424442 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 13:08:47.424711 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:47.424698 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jj6x8\"" Oct 30 13:08:47.518029 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:47.517998 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26kf9\" (UID: \"d03b10bb-ca2a-4636-ad2b-69972fbb56cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9" Oct 30 13:08:47.518236 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:47.518051 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26kf9\" (UID: \"d03b10bb-ca2a-4636-ad2b-69972fbb56cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9" Oct 30 13:08:47.518236 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:47.518089 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26kf9\" (UID: \"d03b10bb-ca2a-4636-ad2b-69972fbb56cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9" Oct 30 13:08:47.518236 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:47.518106 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26kf9\" (UID: \"d03b10bb-ca2a-4636-ad2b-69972fbb56cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9" Oct 30 13:08:47.518236 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:47.518136 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnzdj\" (UniqueName: \"kubernetes.io/projected/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-kube-api-access-mnzdj\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26kf9\" (UID: \"d03b10bb-ca2a-4636-ad2b-69972fbb56cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9" Oct 30 13:08:47.518514 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:47.518281 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26kf9\" (UID: \"d03b10bb-ca2a-4636-ad2b-69972fbb56cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9" Oct 30 13:08:47.518514 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:47.518333 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26kf9\" (UID: \"d03b10bb-ca2a-4636-ad2b-69972fbb56cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9" Oct 30 13:08:47.518989 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:47.518959 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26kf9\" (UID: \"d03b10bb-ca2a-4636-ad2b-69972fbb56cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9" Oct 30 13:08:47.520726 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:47.520701 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26kf9\" (UID: \"d03b10bb-ca2a-4636-ad2b-69972fbb56cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9" Oct 30 13:08:47.520854 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:47.520698 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26kf9\" (UID: \"d03b10bb-ca2a-4636-ad2b-69972fbb56cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9" Oct 30 13:08:47.520854 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:47.520735 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26kf9\" (UID: \"d03b10bb-ca2a-4636-ad2b-69972fbb56cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9" Oct 30 13:08:47.520978 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:47.520922 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26kf9\" (UID: \"d03b10bb-ca2a-4636-ad2b-69972fbb56cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9" Oct 30 13:08:47.544183 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:47.544161 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26kf9\" (UID: \"d03b10bb-ca2a-4636-ad2b-69972fbb56cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9" Oct 30 13:08:47.544461 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:47.544442 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnzdj\" (UniqueName: \"kubernetes.io/projected/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-kube-api-access-mnzdj\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26kf9\" (UID: \"d03b10bb-ca2a-4636-ad2b-69972fbb56cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9" Oct 30 13:08:47.655093 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:47.655020 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9" Oct 30 13:08:47.987054 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:08:47.987023 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd03b10bb_ca2a_4636_ad2b_69972fbb56cf.slice/crio-3e47d4048573b91110d8c677e24ff8433411add7f460d7ed43b4228dbd7b930d WatchSource:0}: Error finding container 3e47d4048573b91110d8c677e24ff8433411add7f460d7ed43b4228dbd7b930d: Status 404 returned error can't find the container with id 3e47d4048573b91110d8c677e24ff8433411add7f460d7ed43b4228dbd7b930d Oct 30 13:08:47.987223 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:47.987201 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9"] Oct 30 13:08:48.911909 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:48.911869 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9" event={"ID":"d03b10bb-ca2a-4636-ad2b-69972fbb56cf","Type":"ContainerStarted","Data":"3e47d4048573b91110d8c677e24ff8433411add7f460d7ed43b4228dbd7b930d"} Oct 30 13:08:51.258618 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:51.258580 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:51.259030 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:51.258655 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:51.944162 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:51.944093 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9" event={"ID":"d03b10bb-ca2a-4636-ad2b-69972fbb56cf","Type":"ContainerStarted","Data":"e91da7006bc29f499d0c4d01085170f6ec6be0845540024581d352a36e29ba09"} Oct 30 13:08:51.944387 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:51.944244 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9" Oct 30 13:08:52.031825 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:52.031761 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9" podStartSLOduration=1.764293747 podStartE2EDuration="5.031743292s" totalImagesPullingTime="3.267449545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:52.014348921 +0000 UTC m=+1430.617847365" watchObservedRunningTime="2026-10-30 13:08:52.031743292 +0000 UTC m=+1430.635241740" Oct 30 13:08:52.954855 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:08:52.954826 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9" Oct 30 13:09:00.053990 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:00.053956 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0132418-n4459"] Oct 30 13:09:00.067689 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:00.067660 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0132418-n4459"] Oct 30 13:09:01.527038 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:01.527002 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d340149-7ccb-4395-a21d-55cdadff7b7b" path="/var/lib/kubelet/pods/8d340149-7ccb-4395-a21d-55cdadff7b7b/volumes" Oct 30 13:09:01.871496 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:01.871420 4931 scope.go:122] "RemoveContainer" containerID="0c5a08b31a7003fee8b00d03db9fc585b25c137ac93eab886b901cb389935a74" Oct 30 13:09:03.861443 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:03.861392 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:09:07.352793 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:09:07.352759 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:09:24.038939 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:24.038896 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-183345e7-z2ghc"] Oct 30 13:09:24.052376 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:24.052344 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-183345e7-z2ghc"] Oct 30 13:09:25.521278 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:25.521233 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="784b2871-f3ab-4cdd-966d-be76d743fa2e" path="/var/lib/kubelet/pods/784b2871-f3ab-4cdd-966d-be76d743fa2e/volumes" Oct 30 13:09:27.536293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:27.536074 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx"] Oct 30 13:09:27.564455 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:27.561658 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:09:27.590531 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:27.590496 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx"] Oct 30 13:09:27.597711 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:27.597683 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-acd125-openshift-default-dockercfg-kct5r\"" Oct 30 13:09:27.700429 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:27.700376 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2721af-325b-407d-8e1e-c1ed692291e8-credential-socket\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:09:27.700625 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:27.700464 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f2721af-325b-407d-8e1e-c1ed692291e8-istiod-ca-cert\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:09:27.700707 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:27.700674 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f2721af-325b-407d-8e1e-c1ed692291e8-istio-data\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:09:27.700803 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:27.700748 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f2721af-325b-407d-8e1e-c1ed692291e8-workload-certs\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:09:27.700928 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:27.700913 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f2721af-325b-407d-8e1e-c1ed692291e8-istio-podinfo\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:09:27.700995 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:27.700984 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f2721af-325b-407d-8e1e-c1ed692291e8-istio-envoy\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:09:27.701036 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:27.701024 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrl4f\" (UniqueName: \"kubernetes.io/projected/0f2721af-325b-407d-8e1e-c1ed692291e8-kube-api-access-xrl4f\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:09:27.701100 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:27.701090 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f2721af-325b-407d-8e1e-c1ed692291e8-istio-token\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:09:27.701135 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:27.701113 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2721af-325b-407d-8e1e-c1ed692291e8-workload-socket\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:09:27.803109 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:27.803003 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f2721af-325b-407d-8e1e-c1ed692291e8-istio-podinfo\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:09:27.803467 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:27.803438 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f2721af-325b-407d-8e1e-c1ed692291e8-istio-envoy\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:09:27.804979 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:27.804948 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrl4f\" (UniqueName: \"kubernetes.io/projected/0f2721af-325b-407d-8e1e-c1ed692291e8-kube-api-access-xrl4f\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:09:27.806056 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:27.806028 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f2721af-325b-407d-8e1e-c1ed692291e8-istio-token\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:09:27.806512 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:27.806491 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2721af-325b-407d-8e1e-c1ed692291e8-workload-socket\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:09:27.806736 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:27.806717 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2721af-325b-407d-8e1e-c1ed692291e8-credential-socket\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:09:27.807579 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:27.807559 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f2721af-325b-407d-8e1e-c1ed692291e8-istiod-ca-cert\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:09:27.807835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:27.807816 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f2721af-325b-407d-8e1e-c1ed692291e8-istio-data\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:09:27.807986 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:27.807969 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f2721af-325b-407d-8e1e-c1ed692291e8-workload-certs\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:09:27.809183 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:27.807190 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2721af-325b-407d-8e1e-c1ed692291e8-credential-socket\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:09:27.810546 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:27.807463 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2721af-325b-407d-8e1e-c1ed692291e8-workload-socket\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:09:27.810732 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:27.808768 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f2721af-325b-407d-8e1e-c1ed692291e8-istiod-ca-cert\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:09:27.810832 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:27.809459 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f2721af-325b-407d-8e1e-c1ed692291e8-workload-certs\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:09:27.810921 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:27.809776 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f2721af-325b-407d-8e1e-c1ed692291e8-istio-envoy\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:09:27.811012 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:27.810002 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f2721af-325b-407d-8e1e-c1ed692291e8-istio-data\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:09:27.830433 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:27.830396 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f2721af-325b-407d-8e1e-c1ed692291e8-istio-podinfo\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:09:27.833868 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:27.833805 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f2721af-325b-407d-8e1e-c1ed692291e8-istio-token\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:09:27.836701 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:27.836669 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrl4f\" (UniqueName: \"kubernetes.io/projected/0f2721af-325b-407d-8e1e-c1ed692291e8-kube-api-access-xrl4f\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:09:27.901707 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:27.901667 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:09:28.112936 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:28.112842 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19c6ea11-sr6hx"] Oct 30 13:09:28.121527 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:28.121494 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19c6ea11-sr6hx"] Oct 30 13:09:28.243761 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:28.243726 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx"] Oct 30 13:09:28.244522 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:09:28.244495 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f2721af_325b_407d_8e1e_c1ed692291e8.slice/crio-d15af90e1aa68bedfe7f54716efc3cb2cc8253ad41e0b32d8c62afac3bb51d15 WatchSource:0}: Error finding container d15af90e1aa68bedfe7f54716efc3cb2cc8253ad41e0b32d8c62afac3bb51d15: Status 404 returned error can't find the container with id d15af90e1aa68bedfe7f54716efc3cb2cc8253ad41e0b32d8c62afac3bb51d15 Oct 30 13:09:28.247203 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:28.247169 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:28.247292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:28.247240 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:28.247334 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:28.247289 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:29.033625 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:29.033587 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27871d28-f4r7w"] Oct 30 13:09:29.046722 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:29.046691 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27871d28-f4r7w"] Oct 30 13:09:29.253841 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:29.253803 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" event={"ID":"0f2721af-325b-407d-8e1e-c1ed692291e8","Type":"ContainerStarted","Data":"ccf15ff02cc32911aaa6ab110724aee34fd6f8c45497292b7c656b93f277e9be"} Oct 30 13:09:29.254043 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:29.253861 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" event={"ID":"0f2721af-325b-407d-8e1e-c1ed692291e8","Type":"ContainerStarted","Data":"d15af90e1aa68bedfe7f54716efc3cb2cc8253ad41e0b32d8c62afac3bb51d15"} Oct 30 13:09:29.313232 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:29.313121 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" podStartSLOduration=2.313103722 podStartE2EDuration="2.313103722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:29.310025207 +0000 UTC m=+1467.913523655" watchObservedRunningTime="2026-10-30 13:09:29.313103722 +0000 UTC m=+1467.916602168" Oct 30 13:09:29.515940 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:29.515905 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5639c55b-e419-4b24-9a26-192e300660df" path="/var/lib/kubelet/pods/5639c55b-e419-4b24-9a26-192e300660df/volumes" Oct 30 13:09:29.516491 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:29.516475 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3463205-9bd3-431e-aa57-c6343263a1b8" path="/var/lib/kubelet/pods/f3463205-9bd3-431e-aa57-c6343263a1b8/volumes" Oct 30 13:09:29.902813 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:29.902781 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:09:29.909054 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:29.909024 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:09:30.261554 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:30.261520 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:09:30.262523 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:30.262503 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:09:34.035293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:34.035239 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e05227f-59b7m"] Oct 30 13:09:34.045903 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:34.045865 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e05227f-59b7m"] Oct 30 13:09:35.517057 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:35.517021 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd9b9df5-4097-42db-94c8-1bbac6681a63" path="/var/lib/kubelet/pods/dd9b9df5-4097-42db-94c8-1bbac6681a63/volumes" Oct 30 13:09:37.039203 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:37.039169 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-739cb84c-l9pfj"] Oct 30 13:09:37.052554 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:37.052522 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-739cb84c-l9pfj"] Oct 30 13:09:37.353776 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:09:37.353685 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:09:37.514600 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:37.514565 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d7d4ac5-af2d-471b-a4c8-53daada47675" path="/var/lib/kubelet/pods/4d7d4ac5-af2d-471b-a4c8-53daada47675/volumes" Oct 30 13:09:37.679387 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:37.677123 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd"] Oct 30 13:09:37.683689 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:37.683659 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:09:37.717726 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:37.717691 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-3e8ee4-openshift-default-dockercfg-hpcvm\"" Oct 30 13:09:37.725225 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:37.725173 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd"] Oct 30 13:09:37.811324 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:37.808382 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/376b917f-91f0-4239-a1b3-eac12620d76e-istio-data\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:09:37.811324 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:37.808478 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/376b917f-91f0-4239-a1b3-eac12620d76e-workload-socket\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:09:37.811324 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:37.808520 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/376b917f-91f0-4239-a1b3-eac12620d76e-istio-podinfo\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:09:37.811324 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:37.808596 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v82l\" (UniqueName: \"kubernetes.io/projected/376b917f-91f0-4239-a1b3-eac12620d76e-kube-api-access-4v82l\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:09:37.811324 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:37.808653 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/376b917f-91f0-4239-a1b3-eac12620d76e-istiod-ca-cert\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:09:37.811324 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:37.808681 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/376b917f-91f0-4239-a1b3-eac12620d76e-workload-certs\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:09:37.811324 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:37.808882 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/376b917f-91f0-4239-a1b3-eac12620d76e-istio-envoy\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:09:37.811324 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:37.808961 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/376b917f-91f0-4239-a1b3-eac12620d76e-istio-token\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:09:37.811324 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:37.808996 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/376b917f-91f0-4239-a1b3-eac12620d76e-credential-socket\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:09:37.912293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:37.911312 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/376b917f-91f0-4239-a1b3-eac12620d76e-istio-envoy\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:09:37.912293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:37.911409 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/376b917f-91f0-4239-a1b3-eac12620d76e-istio-token\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:09:37.912293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:37.911447 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/376b917f-91f0-4239-a1b3-eac12620d76e-credential-socket\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:09:37.912293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:37.911584 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/376b917f-91f0-4239-a1b3-eac12620d76e-istio-data\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:09:37.912293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:37.912033 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/376b917f-91f0-4239-a1b3-eac12620d76e-istio-data\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:09:37.912724 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:37.912588 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/376b917f-91f0-4239-a1b3-eac12620d76e-credential-socket\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:09:37.912724 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:37.912642 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/376b917f-91f0-4239-a1b3-eac12620d76e-workload-socket\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:09:37.912724 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:37.912689 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/376b917f-91f0-4239-a1b3-eac12620d76e-istio-podinfo\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:09:37.912883 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:37.912767 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v82l\" (UniqueName: \"kubernetes.io/projected/376b917f-91f0-4239-a1b3-eac12620d76e-kube-api-access-4v82l\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:09:37.912883 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:37.912825 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/376b917f-91f0-4239-a1b3-eac12620d76e-istiod-ca-cert\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:09:37.912883 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:37.912854 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/376b917f-91f0-4239-a1b3-eac12620d76e-workload-certs\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:09:37.915432 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:37.913351 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/376b917f-91f0-4239-a1b3-eac12620d76e-workload-certs\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:09:37.915432 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:37.913606 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/376b917f-91f0-4239-a1b3-eac12620d76e-workload-socket\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:09:37.915432 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:37.915044 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/376b917f-91f0-4239-a1b3-eac12620d76e-istiod-ca-cert\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:09:37.922253 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:37.922215 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/376b917f-91f0-4239-a1b3-eac12620d76e-istio-envoy\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:09:37.923428 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:37.923399 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/376b917f-91f0-4239-a1b3-eac12620d76e-istio-podinfo\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:09:37.942984 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:37.942945 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v82l\" (UniqueName: \"kubernetes.io/projected/376b917f-91f0-4239-a1b3-eac12620d76e-kube-api-access-4v82l\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:09:37.946242 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:37.945762 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/376b917f-91f0-4239-a1b3-eac12620d76e-istio-token\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:09:38.004317 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:38.004282 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:09:38.352349 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:38.352318 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd"] Oct 30 13:09:38.353712 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:38.353679 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:38.353823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:38.353753 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:38.353823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:38.353782 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:39.343873 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:39.343828 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" event={"ID":"376b917f-91f0-4239-a1b3-eac12620d76e","Type":"ContainerStarted","Data":"f8058e44f1c9d82de9e43e6588266533dc232c20d660fc6cc6cf1dcd4178031c"} Oct 30 13:09:39.344086 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:39.343892 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" event={"ID":"376b917f-91f0-4239-a1b3-eac12620d76e","Type":"ContainerStarted","Data":"1219e3548d46706eda9989fada3812e5611be2208dabe080a7ffab1a5a9a3fe1"} Oct 30 13:09:39.427083 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:39.427026 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" podStartSLOduration=2.427012461 podStartE2EDuration="2.427012461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:39.409885654 +0000 UTC m=+1478.013384100" watchObservedRunningTime="2026-10-30 13:09:39.427012461 +0000 UTC m=+1478.030510907" Oct 30 13:09:40.004777 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:40.004744 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:09:40.010627 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:40.010601 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:09:40.354452 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:40.354378 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:09:40.355354 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:40.355336 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:09:40.648742 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:40.648637 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:09:59.051121 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:59.051083 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a5ac23b-8jlkj"] Oct 30 13:09:59.069169 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:59.069121 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a5ac23b-8jlkj"] Oct 30 13:09:59.515773 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:09:59.515740 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2" path="/var/lib/kubelet/pods/bd76ca70-10f3-4cdb-b1d0-f8b1a34430e2/volumes" Oct 30 13:10:01.728993 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:01.728960 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/2.log" Oct 30 13:10:01.735661 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:01.735636 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/2.log" Oct 30 13:10:01.904838 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:01.904804 4931 scope.go:122] "RemoveContainer" containerID="680b0f28da7a2ca6b8768f69a50369ba702131b27f01c38c61dd50078f9e0db6" Oct 30 13:10:01.915626 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:01.915587 4931 scope.go:122] "RemoveContainer" containerID="b49c78befa805615b8ae23633c69833dea77e4e1ef52f3f486334762422f32b7" Oct 30 13:10:01.925186 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:01.925160 4931 scope.go:122] "RemoveContainer" containerID="f8b7712629e498b801f0af97b15ba0a5716ca190762cffddde09a17a3c2f3a85" Oct 30 13:10:01.935407 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:01.935383 4931 scope.go:122] "RemoveContainer" containerID="3e167b960a71b02331e0ff88b5a79fa35851274c273284bedc45666b75fabc06" Oct 30 13:10:07.355723 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:10:07.355685 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:10:10.450315 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:10.450278 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9077-64c38d"] Oct 30 13:10:10.456455 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:10.456432 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9077-64c38d" Oct 30 13:10:10.471806 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:10.471757 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9077-64c38d"] Oct 30 13:10:10.477098 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:10.477072 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8xhdg\"" Oct 30 13:10:10.586980 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:10.586928 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwhsd\" (UniqueName: \"kubernetes.io/projected/0736227e-c024-482b-9617-a346e0b36820-kube-api-access-fwhsd\") pod \"test-curl-9077-64c38d\" (UID: \"0736227e-c024-482b-9617-a346e0b36820\") " pod="opendatahub/test-curl-9077-64c38d" Oct 30 13:10:10.689132 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:10.689082 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwhsd\" (UniqueName: \"kubernetes.io/projected/0736227e-c024-482b-9617-a346e0b36820-kube-api-access-fwhsd\") pod \"test-curl-9077-64c38d\" (UID: \"0736227e-c024-482b-9617-a346e0b36820\") " pod="opendatahub/test-curl-9077-64c38d" Oct 30 13:10:10.719689 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:10.719616 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwhsd\" (UniqueName: \"kubernetes.io/projected/0736227e-c024-482b-9617-a346e0b36820-kube-api-access-fwhsd\") pod \"test-curl-9077-64c38d\" (UID: \"0736227e-c024-482b-9617-a346e0b36820\") " pod="opendatahub/test-curl-9077-64c38d" Oct 30 13:10:10.774862 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:10.774808 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9077-64c38d" Oct 30 13:10:11.106110 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:11.106079 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9077-64c38d"] Oct 30 13:10:11.106909 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:10:11.106882 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0736227e_c024_482b_9617_a346e0b36820.slice/crio-abf567eb2608b692207afa54b81c0c37c9830c9610f86ede2fcdd06e16259622 WatchSource:0}: Error finding container abf567eb2608b692207afa54b81c0c37c9830c9610f86ede2fcdd06e16259622: Status 404 returned error can't find the container with id abf567eb2608b692207afa54b81c0c37c9830c9610f86ede2fcdd06e16259622 Oct 30 13:10:11.647647 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:11.647608 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9077-64c38d" event={"ID":"0736227e-c024-482b-9617-a346e0b36820","Type":"ContainerStarted","Data":"abf567eb2608b692207afa54b81c0c37c9830c9610f86ede2fcdd06e16259622"} Oct 30 13:10:12.075698 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:12.075644 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md"] Oct 30 13:10:12.075964 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:12.075916 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" podUID="2e8d41cb-daa2-4be3-b42d-90a1951a1fc3" containerName="istio-proxy" containerID="cri-o://960d1dc95143fa3ac61a3addd1f220bf7b0de4ab1b412036f6e99cb1f293d796" gracePeriod=30 Oct 30 13:10:12.603347 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:12.600559 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:10:14.700153 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:14.700114 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9077-64c38d" event={"ID":"0736227e-c024-482b-9617-a346e0b36820","Type":"ContainerStarted","Data":"432e252dd228019aefd1da0c2893c196a6b681b0439ab0988af0e6c5853386ab"} Oct 30 13:10:14.742824 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:14.742763 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9077-64c38d" podStartSLOduration=2.251047587 podStartE2EDuration="4.742747259s" totalImagesPullingTime="2.491699672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:14.741177044 +0000 UTC m=+1513.344675489" watchObservedRunningTime="2026-10-30 13:10:14.742747259 +0000 UTC m=+1513.346245704" Oct 30 13:10:15.710114 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:15.710079 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9077-64c38d"] Oct 30 13:10:15.719787 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:15.717090 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9077-64c38d"] Oct 30 13:10:15.874787 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:15.874749 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9077-924c64"] Oct 30 13:10:15.876594 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:15.876566 4931 cpu_manager.go:414] "containerMap: removing container" podUID="0736227e-c024-482b-9617-a346e0b36820" containerName="test-curl-9077-64c38d" Oct 30 13:10:15.876594 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:15.876590 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0736227e-c024-482b-9617-a346e0b36820" containerName="test-curl-9077-64c38d" Oct 30 13:10:15.876777 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:15.876708 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="0736227e-c024-482b-9617-a346e0b36820" containerName="test-curl-9077-64c38d" Oct 30 13:10:15.880422 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:15.880396 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9077-924c64" Oct 30 13:10:15.895628 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:15.895597 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9077-924c64"] Oct 30 13:10:15.974035 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:15.973949 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm8gx\" (UniqueName: \"kubernetes.io/projected/a7d4526b-0de7-4906-bd09-4cc7cdb74f8e-kube-api-access-vm8gx\") pod \"test-curl-9077-924c64\" (UID: \"a7d4526b-0de7-4906-bd09-4cc7cdb74f8e\") " pod="opendatahub/test-curl-9077-924c64" Oct 30 13:10:16.075563 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:16.075519 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm8gx\" (UniqueName: \"kubernetes.io/projected/a7d4526b-0de7-4906-bd09-4cc7cdb74f8e-kube-api-access-vm8gx\") pod \"test-curl-9077-924c64\" (UID: \"a7d4526b-0de7-4906-bd09-4cc7cdb74f8e\") " pod="opendatahub/test-curl-9077-924c64" Oct 30 13:10:16.100364 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:16.100334 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm8gx\" (UniqueName: \"kubernetes.io/projected/a7d4526b-0de7-4906-bd09-4cc7cdb74f8e-kube-api-access-vm8gx\") pod \"test-curl-9077-924c64\" (UID: \"a7d4526b-0de7-4906-bd09-4cc7cdb74f8e\") " pod="opendatahub/test-curl-9077-924c64" Oct 30 13:10:16.196539 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:16.196502 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9077-924c64" Oct 30 13:10:16.507945 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:16.507914 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9077-924c64"] Oct 30 13:10:16.722370 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:16.722314 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9077-924c64" event={"ID":"a7d4526b-0de7-4906-bd09-4cc7cdb74f8e","Type":"ContainerStarted","Data":"15acf0b83b84c4f9c4a1dab5375274a5786a45b569f71ff2130911ad01d58a2d"} Oct 30 13:10:16.722370 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:16.722363 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9077-924c64" event={"ID":"a7d4526b-0de7-4906-bd09-4cc7cdb74f8e","Type":"ContainerStarted","Data":"671dc986058f3edecc8168f516334f95a21b0a2da3fcbd1bec40e94d0a3c41ec"} Oct 30 13:10:16.722929 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:16.722634 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9077-64c38d" podUID="0736227e-c024-482b-9617-a346e0b36820" containerName="test-curl-9077-64c38d" containerID="cri-o://432e252dd228019aefd1da0c2893c196a6b681b0439ab0988af0e6c5853386ab" gracePeriod=2 Oct 30 13:10:16.764527 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:16.764418 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9077-924c64" podStartSLOduration=1.764403551 podStartE2EDuration="1.764403551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:16.762757833 +0000 UTC m=+1515.366256279" watchObservedRunningTime="2026-10-30 13:10:16.764403551 +0000 UTC m=+1515.367901999" Oct 30 13:10:16.777596 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:10:16.777552 4931 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9077-64c38d\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" podUID="0736227e-c024-482b-9617-a346e0b36820" pod="opendatahub/test-curl-9077-64c38d" Oct 30 13:10:17.353664 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.353593 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" Oct 30 13:10:17.368512 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:10:17.368473 4931 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9077-64c38d\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" podUID="0736227e-c024-482b-9617-a346e0b36820" pod="opendatahub/test-curl-9077-64c38d" Oct 30 13:10:17.497388 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.497351 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-workload-socket\") pod \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " Oct 30 13:10:17.497581 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.497404 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-istio-token\" (UniqueName: \"kubernetes.io/projected/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-istio-token\") pod \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " Oct 30 13:10:17.497581 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.497531 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-workload-certs\") pod \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " Oct 30 13:10:17.497581 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.497573 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-istio-data\") pod \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " Oct 30 13:10:17.497761 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.497605 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-kube-api-access-mvcx4\" (UniqueName: \"kubernetes.io/projected/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-kube-api-access-mvcx4\") pod \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " Oct 30 13:10:17.497761 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.497667 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-workload-socket" pod "2e8d41cb-daa2-4be3-b42d-90a1951a1fc3" (UID: "2e8d41cb-daa2-4be3-b42d-90a1951a1fc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:10:17.497858 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.497762 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-istio-podinfo\") pod \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " Oct 30 13:10:17.497977 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.497871 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-workload-certs" pod "2e8d41cb-daa2-4be3-b42d-90a1951a1fc3" (UID: "2e8d41cb-daa2-4be3-b42d-90a1951a1fc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:10:17.497977 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.497887 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-istio-data" pod "2e8d41cb-daa2-4be3-b42d-90a1951a1fc3" (UID: "2e8d41cb-daa2-4be3-b42d-90a1951a1fc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:10:17.498246 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.498224 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-credential-socket\") pod \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " Oct 30 13:10:17.498892 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.498411 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-istiod-ca-cert\") pod \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " Oct 30 13:10:17.498892 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.498455 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-istio-envoy\") pod \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\" (UID: \"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3\") " Oct 30 13:10:17.498892 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.498607 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-credential-socket" pod "2e8d41cb-daa2-4be3-b42d-90a1951a1fc3" (UID: "2e8d41cb-daa2-4be3-b42d-90a1951a1fc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:10:17.499084 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.498913 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-istiod-ca-cert" pod "2e8d41cb-daa2-4be3-b42d-90a1951a1fc3" (UID: "2e8d41cb-daa2-4be3-b42d-90a1951a1fc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:17.499240 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.499217 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:10:17.499362 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.499247 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:10:17.499362 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.499262 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:10:17.499362 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.499292 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:10:17.499362 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.499304 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:10:17.500262 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.500235 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-istio-podinfo" pod "2e8d41cb-daa2-4be3-b42d-90a1951a1fc3" (UID: "2e8d41cb-daa2-4be3-b42d-90a1951a1fc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:10:17.500262 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.500251 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-istio-token" pod "2e8d41cb-daa2-4be3-b42d-90a1951a1fc3" (UID: "2e8d41cb-daa2-4be3-b42d-90a1951a1fc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:17.500434 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.500412 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-kube-api-access-mvcx4" pod "2e8d41cb-daa2-4be3-b42d-90a1951a1fc3" (UID: "2e8d41cb-daa2-4be3-b42d-90a1951a1fc3"). InnerVolumeSpecName "kube-api-access-mvcx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:17.500934 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.500914 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-istio-envoy" pod "2e8d41cb-daa2-4be3-b42d-90a1951a1fc3" (UID: "2e8d41cb-daa2-4be3-b42d-90a1951a1fc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:10:17.601043 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.601008 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:10:17.601043 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.601041 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:10:17.601258 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.601054 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:10:17.601258 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.601067 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvcx4\" (UniqueName: \"kubernetes.io/projected/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3-kube-api-access-mvcx4\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:10:17.719555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.719505 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9077-924c64"] Oct 30 13:10:17.724017 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.723864 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9077-924c64"] Oct 30 13:10:17.735118 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.735086 4931 generic.go:356] "Generic (PLEG): container finished" podID="2e8d41cb-daa2-4be3-b42d-90a1951a1fc3" containerID="960d1dc95143fa3ac61a3addd1f220bf7b0de4ab1b412036f6e99cb1f293d796" exitCode=0 Oct 30 13:10:17.735317 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.735154 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" Oct 30 13:10:17.735317 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.735176 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" event={"ID":"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3","Type":"ContainerDied","Data":"960d1dc95143fa3ac61a3addd1f220bf7b0de4ab1b412036f6e99cb1f293d796"} Oct 30 13:10:17.735317 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.735291 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md" event={"ID":"2e8d41cb-daa2-4be3-b42d-90a1951a1fc3","Type":"ContainerDied","Data":"7528eeae935e63884fd3e887a8b1e95b305b21c0cc07a3aff8a8cf7408971af7"} Oct 30 13:10:17.735502 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.735327 4931 scope.go:122] "RemoveContainer" containerID="960d1dc95143fa3ac61a3addd1f220bf7b0de4ab1b412036f6e99cb1f293d796" Oct 30 13:10:17.750157 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.750130 4931 scope.go:122] "RemoveContainer" containerID="960d1dc95143fa3ac61a3addd1f220bf7b0de4ab1b412036f6e99cb1f293d796" Oct 30 13:10:17.750507 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:10:17.750484 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"960d1dc95143fa3ac61a3addd1f220bf7b0de4ab1b412036f6e99cb1f293d796\": container with ID starting with 960d1dc95143fa3ac61a3addd1f220bf7b0de4ab1b412036f6e99cb1f293d796 not found: ID does not exist" containerID="960d1dc95143fa3ac61a3addd1f220bf7b0de4ab1b412036f6e99cb1f293d796" Oct 30 13:10:17.750602 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.750519 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"960d1dc95143fa3ac61a3addd1f220bf7b0de4ab1b412036f6e99cb1f293d796"} err="failed to get container status \"960d1dc95143fa3ac61a3addd1f220bf7b0de4ab1b412036f6e99cb1f293d796\": rpc error: code = NotFound desc = could not find container \"960d1dc95143fa3ac61a3addd1f220bf7b0de4ab1b412036f6e99cb1f293d796\": container with ID starting with 960d1dc95143fa3ac61a3addd1f220bf7b0de4ab1b412036f6e99cb1f293d796 not found: ID does not exist" Oct 30 13:10:17.798709 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.798673 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md"] Oct 30 13:10:17.817250 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.817209 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md"] Oct 30 13:10:17.877478 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.877443 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9077-9d64d0"] Oct 30 13:10:17.878099 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.878083 4931 cpu_manager.go:414] "containerMap: removing container" podUID="2e8d41cb-daa2-4be3-b42d-90a1951a1fc3" containerName="istio-proxy" Oct 30 13:10:17.878150 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.878100 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e8d41cb-daa2-4be3-b42d-90a1951a1fc3" containerName="istio-proxy" Oct 30 13:10:17.878185 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.878150 4931 cpu_manager.go:414] "containerMap: removing container" podUID="a7d4526b-0de7-4906-bd09-4cc7cdb74f8e" containerName="test-curl-9077-924c64" Oct 30 13:10:17.878185 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.878155 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7d4526b-0de7-4906-bd09-4cc7cdb74f8e" containerName="test-curl-9077-924c64" Oct 30 13:10:17.878255 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.878246 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7d4526b-0de7-4906-bd09-4cc7cdb74f8e" containerName="test-curl-9077-924c64" Oct 30 13:10:17.878315 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.878260 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e8d41cb-daa2-4be3-b42d-90a1951a1fc3" containerName="istio-proxy" Oct 30 13:10:17.883220 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.883197 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9077-9d64d0" Oct 30 13:10:17.901176 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:17.901106 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9077-9d64d0"] Oct 30 13:10:18.011696 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:18.011607 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r864l\" (UniqueName: \"kubernetes.io/projected/60f6b6d2-be18-4a9b-9d60-48771c4edb6b-kube-api-access-r864l\") pod \"test-curl-9077-9d64d0\" (UID: \"60f6b6d2-be18-4a9b-9d60-48771c4edb6b\") " pod="opendatahub/test-curl-9077-9d64d0" Oct 30 13:10:18.113438 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:18.113401 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r864l\" (UniqueName: \"kubernetes.io/projected/60f6b6d2-be18-4a9b-9d60-48771c4edb6b-kube-api-access-r864l\") pod \"test-curl-9077-9d64d0\" (UID: \"60f6b6d2-be18-4a9b-9d60-48771c4edb6b\") " pod="opendatahub/test-curl-9077-9d64d0" Oct 30 13:10:18.138378 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:18.138343 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r864l\" (UniqueName: \"kubernetes.io/projected/60f6b6d2-be18-4a9b-9d60-48771c4edb6b-kube-api-access-r864l\") pod \"test-curl-9077-9d64d0\" (UID: \"60f6b6d2-be18-4a9b-9d60-48771c4edb6b\") " pod="opendatahub/test-curl-9077-9d64d0" Oct 30 13:10:18.196506 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:18.196471 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9077-9d64d0" Oct 30 13:10:18.534750 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:10:18.534707 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60f6b6d2_be18_4a9b_9d60_48771c4edb6b.slice/crio-f686b6aca2d0c59c1efe7827f4ab0ea205dce099148c3922db392a60c16d595f WatchSource:0}: Error finding container f686b6aca2d0c59c1efe7827f4ab0ea205dce099148c3922db392a60c16d595f: Status 404 returned error can't find the container with id f686b6aca2d0c59c1efe7827f4ab0ea205dce099148c3922db392a60c16d595f Oct 30 13:10:18.534902 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:18.534824 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9077-9d64d0"] Oct 30 13:10:18.754365 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:18.754245 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9077-924c64" podUID="a7d4526b-0de7-4906-bd09-4cc7cdb74f8e" containerName="test-curl-9077-924c64" containerID="cri-o://15acf0b83b84c4f9c4a1dab5375274a5786a45b569f71ff2130911ad01d58a2d" gracePeriod=2 Oct 30 13:10:18.755458 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:18.755401 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9077-9d64d0" event={"ID":"60f6b6d2-be18-4a9b-9d60-48771c4edb6b","Type":"ContainerStarted","Data":"ed0e04b5619dc175caa2f9ef6e13fc7c954f80fcc66a69f2d6cecb1e33bba9b4"} Oct 30 13:10:18.755458 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:18.755458 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9077-9d64d0" event={"ID":"60f6b6d2-be18-4a9b-9d60-48771c4edb6b","Type":"ContainerStarted","Data":"f686b6aca2d0c59c1efe7827f4ab0ea205dce099148c3922db392a60c16d595f"} Oct 30 13:10:18.800645 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:18.800590 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9077-9d64d0" podStartSLOduration=1.800573682 podStartE2EDuration="1.800573682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:18.79662104 +0000 UTC m=+1517.400119482" watchObservedRunningTime="2026-10-30 13:10:18.800573682 +0000 UTC m=+1517.404072127" Oct 30 13:10:18.811452 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:10:18.811403 4931 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9077-924c64\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" podUID="a7d4526b-0de7-4906-bd09-4cc7cdb74f8e" pod="opendatahub/test-curl-9077-924c64" Oct 30 13:10:18.826077 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:10:18.826030 4931 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9077-924c64\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" podUID="a7d4526b-0de7-4906-bd09-4cc7cdb74f8e" pod="opendatahub/test-curl-9077-924c64" Oct 30 13:10:18.955083 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:18.955057 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9077-64c38d" Oct 30 13:10:18.970153 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:10:18.970117 4931 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9077-924c64\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" podUID="a7d4526b-0de7-4906-bd09-4cc7cdb74f8e" pod="opendatahub/test-curl-9077-924c64" Oct 30 13:10:19.032688 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:19.032656 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0736227e-c024-482b-9617-a346e0b36820-kube-api-access-fwhsd\" (UniqueName: \"kubernetes.io/projected/0736227e-c024-482b-9617-a346e0b36820-kube-api-access-fwhsd\") pod \"0736227e-c024-482b-9617-a346e0b36820\" (UID: \"0736227e-c024-482b-9617-a346e0b36820\") " Oct 30 13:10:19.034841 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:19.034810 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0736227e-c024-482b-9617-a346e0b36820-kube-api-access-fwhsd" pod "0736227e-c024-482b-9617-a346e0b36820" (UID: "0736227e-c024-482b-9617-a346e0b36820"). InnerVolumeSpecName "kube-api-access-fwhsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:19.134613 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:19.134525 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwhsd\" (UniqueName: \"kubernetes.io/projected/0736227e-c024-482b-9617-a346e0b36820-kube-api-access-fwhsd\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:10:19.518499 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:19.518457 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0736227e-c024-482b-9617-a346e0b36820" path="/var/lib/kubelet/pods/0736227e-c024-482b-9617-a346e0b36820/volumes" Oct 30 13:10:19.519261 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:19.519241 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e8d41cb-daa2-4be3-b42d-90a1951a1fc3" path="/var/lib/kubelet/pods/2e8d41cb-daa2-4be3-b42d-90a1951a1fc3/volumes" Oct 30 13:10:19.811027 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:19.810944 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9077-9d64d0"] Oct 30 13:10:19.811546 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:19.811288 4931 generic.go:356] "Generic (PLEG): container finished" podID="0736227e-c024-482b-9617-a346e0b36820" containerID="432e252dd228019aefd1da0c2893c196a6b681b0439ab0988af0e6c5853386ab" exitCode=137 Oct 30 13:10:19.811546 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:19.811478 4931 scope.go:122] "RemoveContainer" containerID="432e252dd228019aefd1da0c2893c196a6b681b0439ab0988af0e6c5853386ab" Oct 30 13:10:19.811751 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:19.811731 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9077-64c38d" Oct 30 13:10:19.817382 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:19.817352 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9077-9d64d0"] Oct 30 13:10:19.824333 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:19.824311 4931 scope.go:122] "RemoveContainer" containerID="432e252dd228019aefd1da0c2893c196a6b681b0439ab0988af0e6c5853386ab" Oct 30 13:10:19.824674 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:10:19.824643 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"432e252dd228019aefd1da0c2893c196a6b681b0439ab0988af0e6c5853386ab\": container with ID starting with 432e252dd228019aefd1da0c2893c196a6b681b0439ab0988af0e6c5853386ab not found: ID does not exist" containerID="432e252dd228019aefd1da0c2893c196a6b681b0439ab0988af0e6c5853386ab" Oct 30 13:10:19.824777 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:19.824684 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"432e252dd228019aefd1da0c2893c196a6b681b0439ab0988af0e6c5853386ab"} err="failed to get container status \"432e252dd228019aefd1da0c2893c196a6b681b0439ab0988af0e6c5853386ab\": rpc error: code = NotFound desc = could not find container \"432e252dd228019aefd1da0c2893c196a6b681b0439ab0988af0e6c5853386ab\": container with ID starting with 432e252dd228019aefd1da0c2893c196a6b681b0439ab0988af0e6c5853386ab not found: ID does not exist" Oct 30 13:10:20.823075 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:20.823034 4931 generic.go:356] "Generic (PLEG): container finished" podID="a7d4526b-0de7-4906-bd09-4cc7cdb74f8e" containerID="15acf0b83b84c4f9c4a1dab5375274a5786a45b569f71ff2130911ad01d58a2d" exitCode=137 Oct 30 13:10:20.823529 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:20.823432 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9077-9d64d0" podUID="60f6b6d2-be18-4a9b-9d60-48771c4edb6b" containerName="test-curl-9077-9d64d0" containerID="cri-o://ed0e04b5619dc175caa2f9ef6e13fc7c954f80fcc66a69f2d6cecb1e33bba9b4" gracePeriod=2 Oct 30 13:10:20.838435 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:10:20.838399 4931 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9077-9d64d0\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" podUID="60f6b6d2-be18-4a9b-9d60-48771c4edb6b" pod="opendatahub/test-curl-9077-9d64d0" Oct 30 13:10:20.972442 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:20.972415 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9077-924c64" Oct 30 13:10:20.988403 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:10:20.988358 4931 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9077-9d64d0\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" podUID="60f6b6d2-be18-4a9b-9d60-48771c4edb6b" pod="opendatahub/test-curl-9077-9d64d0" Oct 30 13:10:21.062841 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:21.062745 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7d4526b-0de7-4906-bd09-4cc7cdb74f8e-kube-api-access-vm8gx\" (UniqueName: \"kubernetes.io/projected/a7d4526b-0de7-4906-bd09-4cc7cdb74f8e-kube-api-access-vm8gx\") pod \"a7d4526b-0de7-4906-bd09-4cc7cdb74f8e\" (UID: \"a7d4526b-0de7-4906-bd09-4cc7cdb74f8e\") " Oct 30 13:10:21.064921 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:21.064887 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d4526b-0de7-4906-bd09-4cc7cdb74f8e-kube-api-access-vm8gx" pod "a7d4526b-0de7-4906-bd09-4cc7cdb74f8e" (UID: "a7d4526b-0de7-4906-bd09-4cc7cdb74f8e"). InnerVolumeSpecName "kube-api-access-vm8gx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:21.165777 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:21.165743 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm8gx\" (UniqueName: \"kubernetes.io/projected/a7d4526b-0de7-4906-bd09-4cc7cdb74f8e-kube-api-access-vm8gx\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:10:21.523465 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:21.523428 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7d4526b-0de7-4906-bd09-4cc7cdb74f8e" path="/var/lib/kubelet/pods/a7d4526b-0de7-4906-bd09-4cc7cdb74f8e/volumes" Oct 30 13:10:21.530215 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:10:21.530147 4931 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9077-9d64d0\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" podUID="60f6b6d2-be18-4a9b-9d60-48771c4edb6b" pod="opendatahub/test-curl-9077-9d64d0" Oct 30 13:10:21.833398 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:21.833315 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9077-924c64" Oct 30 13:10:21.833878 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:21.833314 4931 scope.go:122] "RemoveContainer" containerID="15acf0b83b84c4f9c4a1dab5375274a5786a45b569f71ff2130911ad01d58a2d" Oct 30 13:10:23.055107 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:23.055078 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9077-9d64d0" Oct 30 13:10:23.095150 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:23.095056 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60f6b6d2-be18-4a9b-9d60-48771c4edb6b-kube-api-access-r864l\" (UniqueName: \"kubernetes.io/projected/60f6b6d2-be18-4a9b-9d60-48771c4edb6b-kube-api-access-r864l\") pod \"60f6b6d2-be18-4a9b-9d60-48771c4edb6b\" (UID: \"60f6b6d2-be18-4a9b-9d60-48771c4edb6b\") " Oct 30 13:10:23.097326 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:23.097293 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60f6b6d2-be18-4a9b-9d60-48771c4edb6b-kube-api-access-r864l" pod "60f6b6d2-be18-4a9b-9d60-48771c4edb6b" (UID: "60f6b6d2-be18-4a9b-9d60-48771c4edb6b"). InnerVolumeSpecName "kube-api-access-r864l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:23.197099 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:23.197059 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r864l\" (UniqueName: \"kubernetes.io/projected/60f6b6d2-be18-4a9b-9d60-48771c4edb6b-kube-api-access-r864l\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:10:23.524327 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:23.521198 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60f6b6d2-be18-4a9b-9d60-48771c4edb6b" path="/var/lib/kubelet/pods/60f6b6d2-be18-4a9b-9d60-48771c4edb6b/volumes" Oct 30 13:10:23.702620 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:23.702580 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4189e66-j5hfv"] Oct 30 13:10:23.703318 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:23.703296 4931 cpu_manager.go:414] "containerMap: removing container" podUID="60f6b6d2-be18-4a9b-9d60-48771c4edb6b" containerName="test-curl-9077-9d64d0" Oct 30 13:10:23.703318 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:23.703318 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60f6b6d2-be18-4a9b-9d60-48771c4edb6b" containerName="test-curl-9077-9d64d0" Oct 30 13:10:23.703565 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:23.703550 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="60f6b6d2-be18-4a9b-9d60-48771c4edb6b" containerName="test-curl-9077-9d64d0" Oct 30 13:10:23.707458 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:23.707420 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4189e66-j5hfv" Oct 30 13:10:23.726654 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:23.726620 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4189e66-j5hfv"] Oct 30 13:10:23.735284 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:23.735224 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:10:23.735464 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:23.735296 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:10:23.806795 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:23.806700 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhfzd\" (UniqueName: \"kubernetes.io/projected/66dae659-dcb6-4bd9-97f6-8ec85c68dc8e-kube-api-access-mhfzd\") pod \"maas-api-revoke-sub-f4189e66-j5hfv\" (UID: \"66dae659-dcb6-4bd9-97f6-8ec85c68dc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4189e66-j5hfv" Oct 30 13:10:23.806795 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:23.806779 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66dae659-dcb6-4bd9-97f6-8ec85c68dc8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4189e66-j5hfv\" (UID: \"66dae659-dcb6-4bd9-97f6-8ec85c68dc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4189e66-j5hfv" Oct 30 13:10:23.806969 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:23.806859 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66dae659-dcb6-4bd9-97f6-8ec85c68dc8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4189e66-j5hfv\" (UID: \"66dae659-dcb6-4bd9-97f6-8ec85c68dc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4189e66-j5hfv" Oct 30 13:10:23.853747 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:23.853708 4931 generic.go:356] "Generic (PLEG): container finished" podID="60f6b6d2-be18-4a9b-9d60-48771c4edb6b" containerID="ed0e04b5619dc175caa2f9ef6e13fc7c954f80fcc66a69f2d6cecb1e33bba9b4" exitCode=137 Oct 30 13:10:23.853939 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:23.853761 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9077-9d64d0" Oct 30 13:10:23.853939 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:23.853770 4931 scope.go:122] "RemoveContainer" containerID="ed0e04b5619dc175caa2f9ef6e13fc7c954f80fcc66a69f2d6cecb1e33bba9b4" Oct 30 13:10:23.864568 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:23.864547 4931 scope.go:122] "RemoveContainer" containerID="ed0e04b5619dc175caa2f9ef6e13fc7c954f80fcc66a69f2d6cecb1e33bba9b4" Oct 30 13:10:23.864916 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:10:23.864897 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed0e04b5619dc175caa2f9ef6e13fc7c954f80fcc66a69f2d6cecb1e33bba9b4\": container with ID starting with ed0e04b5619dc175caa2f9ef6e13fc7c954f80fcc66a69f2d6cecb1e33bba9b4 not found: ID does not exist" containerID="ed0e04b5619dc175caa2f9ef6e13fc7c954f80fcc66a69f2d6cecb1e33bba9b4" Oct 30 13:10:23.864972 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:23.864926 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed0e04b5619dc175caa2f9ef6e13fc7c954f80fcc66a69f2d6cecb1e33bba9b4"} err="failed to get container status \"ed0e04b5619dc175caa2f9ef6e13fc7c954f80fcc66a69f2d6cecb1e33bba9b4\": rpc error: code = NotFound desc = could not find container \"ed0e04b5619dc175caa2f9ef6e13fc7c954f80fcc66a69f2d6cecb1e33bba9b4\": container with ID starting with ed0e04b5619dc175caa2f9ef6e13fc7c954f80fcc66a69f2d6cecb1e33bba9b4 not found: ID does not exist" Oct 30 13:10:23.908351 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:23.908305 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhfzd\" (UniqueName: \"kubernetes.io/projected/66dae659-dcb6-4bd9-97f6-8ec85c68dc8e-kube-api-access-mhfzd\") pod \"maas-api-revoke-sub-f4189e66-j5hfv\" (UID: \"66dae659-dcb6-4bd9-97f6-8ec85c68dc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4189e66-j5hfv" Oct 30 13:10:23.908527 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:23.908383 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66dae659-dcb6-4bd9-97f6-8ec85c68dc8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4189e66-j5hfv\" (UID: \"66dae659-dcb6-4bd9-97f6-8ec85c68dc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4189e66-j5hfv" Oct 30 13:10:23.908527 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:23.908424 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66dae659-dcb6-4bd9-97f6-8ec85c68dc8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4189e66-j5hfv\" (UID: \"66dae659-dcb6-4bd9-97f6-8ec85c68dc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4189e66-j5hfv" Oct 30 13:10:23.909169 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:23.909148 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66dae659-dcb6-4bd9-97f6-8ec85c68dc8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4189e66-j5hfv\" (UID: \"66dae659-dcb6-4bd9-97f6-8ec85c68dc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4189e66-j5hfv" Oct 30 13:10:23.910823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:23.910792 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66dae659-dcb6-4bd9-97f6-8ec85c68dc8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4189e66-j5hfv\" (UID: \"66dae659-dcb6-4bd9-97f6-8ec85c68dc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4189e66-j5hfv" Oct 30 13:10:23.934341 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:23.934304 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhfzd\" (UniqueName: \"kubernetes.io/projected/66dae659-dcb6-4bd9-97f6-8ec85c68dc8e-kube-api-access-mhfzd\") pod \"maas-api-revoke-sub-f4189e66-j5hfv\" (UID: \"66dae659-dcb6-4bd9-97f6-8ec85c68dc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4189e66-j5hfv" Oct 30 13:10:24.021526 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:24.021488 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4189e66-j5hfv" Oct 30 13:10:24.333067 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:24.333033 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4189e66-j5hfv"] Oct 30 13:10:24.334337 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:10:24.334260 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66dae659_dcb6_4bd9_97f6_8ec85c68dc8e.slice/crio-f426a19419327c720a744ebe3d0d34f55aa9f0443725432546da94a3a734afdf WatchSource:0}: Error finding container f426a19419327c720a744ebe3d0d34f55aa9f0443725432546da94a3a734afdf: Status 404 returned error can't find the container with id f426a19419327c720a744ebe3d0d34f55aa9f0443725432546da94a3a734afdf Oct 30 13:10:24.864694 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:24.864651 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4189e66-j5hfv" event={"ID":"66dae659-dcb6-4bd9-97f6-8ec85c68dc8e","Type":"ContainerStarted","Data":"ac2b82f085dc7c61b3863c64fedeeb7cc0bc47953be3829ff19a7e871ddb2673"} Oct 30 13:10:24.864880 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:24.864747 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4189e66-j5hfv" event={"ID":"66dae659-dcb6-4bd9-97f6-8ec85c68dc8e","Type":"ContainerStarted","Data":"f426a19419327c720a744ebe3d0d34f55aa9f0443725432546da94a3a734afdf"} Oct 30 13:10:24.909239 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:24.909130 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4189e66-j5hfv" podStartSLOduration=1.909115967 podStartE2EDuration="1.909115967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:24.907696267 +0000 UTC m=+1523.511194714" watchObservedRunningTime="2026-10-30 13:10:24.909115967 +0000 UTC m=+1523.512614412" Oct 30 13:10:25.874040 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:25.874006 4931 generic.go:356] "Generic (PLEG): container finished" podID="66dae659-dcb6-4bd9-97f6-8ec85c68dc8e" containerID="ac2b82f085dc7c61b3863c64fedeeb7cc0bc47953be3829ff19a7e871ddb2673" exitCode=0 Oct 30 13:10:25.874503 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:25.874099 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4189e66-j5hfv" event={"ID":"66dae659-dcb6-4bd9-97f6-8ec85c68dc8e","Type":"ContainerDied","Data":"ac2b82f085dc7c61b3863c64fedeeb7cc0bc47953be3829ff19a7e871ddb2673"} Oct 30 13:10:27.105102 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:27.105073 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4189e66-j5hfv" Oct 30 13:10:27.273033 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:27.272994 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66dae659-dcb6-4bd9-97f6-8ec85c68dc8e-kube-api-access-mhfzd\" (UniqueName: \"kubernetes.io/projected/66dae659-dcb6-4bd9-97f6-8ec85c68dc8e-kube-api-access-mhfzd\") pod \"66dae659-dcb6-4bd9-97f6-8ec85c68dc8e\" (UID: \"66dae659-dcb6-4bd9-97f6-8ec85c68dc8e\") " Oct 30 13:10:27.273215 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:27.273200 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66dae659-dcb6-4bd9-97f6-8ec85c68dc8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66dae659-dcb6-4bd9-97f6-8ec85c68dc8e-maas-api-xks-ca\") pod \"66dae659-dcb6-4bd9-97f6-8ec85c68dc8e\" (UID: \"66dae659-dcb6-4bd9-97f6-8ec85c68dc8e\") " Oct 30 13:10:27.273352 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:27.273338 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66dae659-dcb6-4bd9-97f6-8ec85c68dc8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66dae659-dcb6-4bd9-97f6-8ec85c68dc8e-maas-api-openshift-ca\") pod \"66dae659-dcb6-4bd9-97f6-8ec85c68dc8e\" (UID: \"66dae659-dcb6-4bd9-97f6-8ec85c68dc8e\") " Oct 30 13:10:27.273701 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:27.273676 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66dae659-dcb6-4bd9-97f6-8ec85c68dc8e-maas-api-openshift-ca" pod "66dae659-dcb6-4bd9-97f6-8ec85c68dc8e" (UID: "66dae659-dcb6-4bd9-97f6-8ec85c68dc8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:27.273910 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:27.273889 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66dae659-dcb6-4bd9-97f6-8ec85c68dc8e-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:10:27.275235 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:27.275205 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66dae659-dcb6-4bd9-97f6-8ec85c68dc8e-kube-api-access-mhfzd" pod "66dae659-dcb6-4bd9-97f6-8ec85c68dc8e" (UID: "66dae659-dcb6-4bd9-97f6-8ec85c68dc8e"). InnerVolumeSpecName "kube-api-access-mhfzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:27.275426 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:27.275309 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66dae659-dcb6-4bd9-97f6-8ec85c68dc8e-maas-api-xks-ca" pod "66dae659-dcb6-4bd9-97f6-8ec85c68dc8e" (UID: "66dae659-dcb6-4bd9-97f6-8ec85c68dc8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:10:27.375740 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:27.375695 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66dae659-dcb6-4bd9-97f6-8ec85c68dc8e-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:10:27.375740 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:27.375728 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhfzd\" (UniqueName: \"kubernetes.io/projected/66dae659-dcb6-4bd9-97f6-8ec85c68dc8e-kube-api-access-mhfzd\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:10:27.890314 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:27.890212 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4189e66-j5hfv" Oct 30 13:10:27.890314 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:27.890244 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4189e66-j5hfv" event={"ID":"66dae659-dcb6-4bd9-97f6-8ec85c68dc8e","Type":"ContainerDied","Data":"f426a19419327c720a744ebe3d0d34f55aa9f0443725432546da94a3a734afdf"} Oct 30 13:10:27.890546 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:27.890328 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f426a19419327c720a744ebe3d0d34f55aa9f0443725432546da94a3a734afdf" Oct 30 13:10:37.381725 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:37.380395 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" podUID="f3b342db-9a01-46ad-b476-bfc912d9ce1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:37.404746 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:37.404691 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" podUID="0c5a9f00-6d5b-4860-a210-2fec24afb322" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:37.551903 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:10:37.551860 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:10:37.597484 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:37.597432 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" podUID="606b252b-881c-4d40-8ba9-fc4ed937a995" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 13:10:37.655340 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:37.655190 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" podUID="9577f571-30cd-4043-a3e4-72bf95837552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:37.751431 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:37.751368 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" podUID="089cb620-913d-47c7-b8f7-9b2298c194bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:37.869426 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:37.869367 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" podUID="fb4d115a-ce18-42fe-a26e-d28f0cbadd1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:37.932493 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:37.932373 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" podUID="ee8505de-cac0-4e49-bd15-a044536b10e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:38.003007 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:38.002955 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" podUID="7ad1989a-c73f-4647-9e50-855605bdb08e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:38.159431 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:38.159368 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" podUID="5a504237-e5a3-491f-8f84-0419cf502f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:38.211292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:38.207955 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" podUID="64468005-61ea-4f2a-b72c-9c30c6a4d686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:38.411296 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:38.410729 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" podUID="0c5a9f00-6d5b-4860-a210-2fec24afb322" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:38.411835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:38.411477 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 13:10:38.459052 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:38.457891 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" podUID="65799c14-8de4-4b4a-90df-7cd257e1e581" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:38.687488 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:38.683401 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" podUID="9577f571-30cd-4043-a3e4-72bf95837552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:38.687488 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:38.683941 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:10:38.744478 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:38.744433 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" podUID="606b252b-881c-4d40-8ba9-fc4ed937a995" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:38.745142 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:38.745115 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 13:10:38.758553 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:38.758509 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" podUID="089cb620-913d-47c7-b8f7-9b2298c194bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:38.771175 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:38.759787 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 13:10:38.781295 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:38.777792 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" podUID="b1e86eea-4824-4c50-8ad3-2a15c3365c3d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:38.889291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:38.888759 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" podUID="fb4d115a-ce18-42fe-a26e-d28f0cbadd1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded" Oct 30 13:10:38.889291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:38.889184 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" Oct 30 13:10:38.935423 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:38.935371 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" podUID="ee8505de-cac0-4e49-bd15-a044536b10e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:38.974290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:38.965356 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" Oct 30 13:10:39.079289 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:39.028337 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" podUID="7ad1989a-c73f-4647-9e50-855605bdb08e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:39.079289 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:39.028785 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:10:39.187292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:39.163654 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" podUID="5a504237-e5a3-491f-8f84-0419cf502f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:39.256392 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:39.189718 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:10:39.305526 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:39.253117 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" podUID="64468005-61ea-4f2a-b72c-9c30c6a4d686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:39.307936 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:39.307795 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 13:10:39.433025 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:39.432974 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" podUID="0c5a9f00-6d5b-4860-a210-2fec24afb322" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 13:10:39.455832 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:39.437025 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" podUID="e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 13:10:39.549413 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:39.532405 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" podUID="65799c14-8de4-4b4a-90df-7cd257e1e581" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:39.637293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:39.617092 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:10:39.805354 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:39.804794 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" podUID="606b252b-881c-4d40-8ba9-fc4ed937a995" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 13:10:39.809110 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:39.809071 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" podUID="089cb620-913d-47c7-b8f7-9b2298c194bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 13:10:39.816133 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:39.814643 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" podUID="b1e86eea-4824-4c50-8ad3-2a15c3365c3d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:39.817790 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:39.814681 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" podUID="9577f571-30cd-4043-a3e4-72bf95837552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 13:10:39.817790 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:39.816805 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" Oct 30 13:10:39.925650 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:39.918310 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" podUID="fb4d115a-ce18-42fe-a26e-d28f0cbadd1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded" Oct 30 13:10:40.006969 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:40.003521 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" podUID="ee8505de-cac0-4e49-bd15-a044536b10e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:40.032756 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:40.032713 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" podUID="7ad1989a-c73f-4647-9e50-855605bdb08e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:40.213752 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:40.213705 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" podUID="5a504237-e5a3-491f-8f84-0419cf502f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 13:10:40.334691 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:40.334641 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" podUID="64468005-61ea-4f2a-b72c-9c30c6a4d686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:40.437327 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:40.434372 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" podUID="0c5a9f00-6d5b-4860-a210-2fec24afb322" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:40.537485 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:40.537375 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" podUID="65799c14-8de4-4b4a-90df-7cd257e1e581" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:40.603148 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:40.603116 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" Oct 30 13:10:40.714626 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:40.714591 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:10:40.731688 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:40.729497 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:10:40.786085 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:40.786053 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 13:10:40.809584 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:40.809476 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" podUID="606b252b-881c-4d40-8ba9-fc4ed937a995" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:40.817769 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:40.817720 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" podUID="b1e86eea-4824-4c50-8ad3-2a15c3365c3d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:40.818149 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:40.818031 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" podUID="089cb620-913d-47c7-b8f7-9b2298c194bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:40.829313 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:40.828366 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" podUID="9577f571-30cd-4043-a3e4-72bf95837552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:40.924536 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:40.923704 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" podUID="fb4d115a-ce18-42fe-a26e-d28f0cbadd1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:41.024658 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:41.023186 4931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" podUID="606b252b-881c-4d40-8ba9-fc4ed937a995" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:41.024658 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:41.023817 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 13:10:41.106688 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:41.106579 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:10:41.125860 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:41.120003 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 13:10:41.163764 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:41.163733 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" Oct 30 13:10:41.373737 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:41.369221 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:10:41.424716 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:41.424664 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" podUID="e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:41.722314 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:41.722285 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" Oct 30 13:10:41.793604 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:41.793567 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 13:10:42.214362 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:42.214324 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn"] Oct 30 13:10:42.214615 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:42.214584 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" podUID="3680a3d2-0b13-40df-aeeb-6a924495f8d2" containerName="istio-proxy" containerID="cri-o://9bce9341e990823fb376085bb78badbdae08554f3cfa22fa99036200228d18b0" gracePeriod=30 Oct 30 13:10:42.221168 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:42.221143 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4ec3f23-h7q9k"] Oct 30 13:10:42.222584 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:42.222564 4931 cpu_manager.go:414] "containerMap: removing container" podUID="66dae659-dcb6-4bd9-97f6-8ec85c68dc8e" containerName="revoke-keys" Oct 30 13:10:42.222739 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:42.222719 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66dae659-dcb6-4bd9-97f6-8ec85c68dc8e" containerName="revoke-keys" Oct 30 13:10:42.223032 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:42.223021 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="66dae659-dcb6-4bd9-97f6-8ec85c68dc8e" containerName="revoke-keys" Oct 30 13:10:42.243772 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:42.243742 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4ec3f23-h7q9k"] Oct 30 13:10:42.311294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:42.245407 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ec3f23-h7q9k" Oct 30 13:10:42.326056 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:42.326027 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:10:42.326236 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:42.326218 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:10:42.402980 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:42.402939 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/519b9db0-c559-4f08-be61-ea559d8a1189-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4ec3f23-h7q9k\" (UID: \"519b9db0-c559-4f08-be61-ea559d8a1189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ec3f23-h7q9k" Oct 30 13:10:42.403315 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:42.403288 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/519b9db0-c559-4f08-be61-ea559d8a1189-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4ec3f23-h7q9k\" (UID: \"519b9db0-c559-4f08-be61-ea559d8a1189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ec3f23-h7q9k" Oct 30 13:10:42.403553 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:42.403536 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q82q6\" (UniqueName: \"kubernetes.io/projected/519b9db0-c559-4f08-be61-ea559d8a1189-kube-api-access-q82q6\") pod \"maas-api-revoke-sub-c4ec3f23-h7q9k\" (UID: \"519b9db0-c559-4f08-be61-ea559d8a1189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ec3f23-h7q9k" Oct 30 13:10:42.505480 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:42.505382 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/519b9db0-c559-4f08-be61-ea559d8a1189-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4ec3f23-h7q9k\" (UID: \"519b9db0-c559-4f08-be61-ea559d8a1189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ec3f23-h7q9k" Oct 30 13:10:42.505480 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:42.505467 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/519b9db0-c559-4f08-be61-ea559d8a1189-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4ec3f23-h7q9k\" (UID: \"519b9db0-c559-4f08-be61-ea559d8a1189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ec3f23-h7q9k" Oct 30 13:10:42.505746 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:42.505570 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q82q6\" (UniqueName: \"kubernetes.io/projected/519b9db0-c559-4f08-be61-ea559d8a1189-kube-api-access-q82q6\") pod \"maas-api-revoke-sub-c4ec3f23-h7q9k\" (UID: \"519b9db0-c559-4f08-be61-ea559d8a1189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ec3f23-h7q9k" Oct 30 13:10:42.506727 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:42.506693 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/519b9db0-c559-4f08-be61-ea559d8a1189-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4ec3f23-h7q9k\" (UID: \"519b9db0-c559-4f08-be61-ea559d8a1189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ec3f23-h7q9k" Oct 30 13:10:42.508712 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:42.508683 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/519b9db0-c559-4f08-be61-ea559d8a1189-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4ec3f23-h7q9k\" (UID: \"519b9db0-c559-4f08-be61-ea559d8a1189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ec3f23-h7q9k" Oct 30 13:10:42.532140 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:42.532107 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q82q6\" (UniqueName: \"kubernetes.io/projected/519b9db0-c559-4f08-be61-ea559d8a1189-kube-api-access-q82q6\") pod \"maas-api-revoke-sub-c4ec3f23-h7q9k\" (UID: \"519b9db0-c559-4f08-be61-ea559d8a1189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ec3f23-h7q9k" Oct 30 13:10:42.626975 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:42.626938 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ec3f23-h7q9k" Oct 30 13:10:43.096326 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:43.096296 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4ec3f23-h7q9k"] Oct 30 13:10:43.101349 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:10:43.101308 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod519b9db0_c559_4f08_be61_ea559d8a1189.slice/crio-5ad1fada40ceec17432933cdca340e62b7e5df120411f7c5ba33df99591cbe5d WatchSource:0}: Error finding container 5ad1fada40ceec17432933cdca340e62b7e5df120411f7c5ba33df99591cbe5d: Status 404 returned error can't find the container with id 5ad1fada40ceec17432933cdca340e62b7e5df120411f7c5ba33df99591cbe5d Oct 30 13:10:43.657491 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:43.655956 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ec3f23-h7q9k" event={"ID":"519b9db0-c559-4f08-be61-ea559d8a1189","Type":"ContainerStarted","Data":"5ad1fada40ceec17432933cdca340e62b7e5df120411f7c5ba33df99591cbe5d"} Oct 30 13:10:44.670946 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:44.670899 4931 generic.go:356] "Generic (PLEG): container finished" podID="519b9db0-c559-4f08-be61-ea559d8a1189" containerID="e5f435be596b2cc97070d05e2b507610ef8b66b49e3352c680eb42841a60bbd8" exitCode=0 Oct 30 13:10:44.671534 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:44.671229 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ec3f23-h7q9k" event={"ID":"519b9db0-c559-4f08-be61-ea559d8a1189","Type":"ContainerDied","Data":"e5f435be596b2cc97070d05e2b507610ef8b66b49e3352c680eb42841a60bbd8"} Oct 30 13:10:45.911092 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:45.910800 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ec3f23-h7q9k" Oct 30 13:10:45.959625 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:45.959586 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/519b9db0-c559-4f08-be61-ea559d8a1189-kube-api-access-q82q6\" (UniqueName: \"kubernetes.io/projected/519b9db0-c559-4f08-be61-ea559d8a1189-kube-api-access-q82q6\") pod \"519b9db0-c559-4f08-be61-ea559d8a1189\" (UID: \"519b9db0-c559-4f08-be61-ea559d8a1189\") " Oct 30 13:10:45.959964 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:45.959705 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/519b9db0-c559-4f08-be61-ea559d8a1189-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/519b9db0-c559-4f08-be61-ea559d8a1189-maas-api-openshift-ca\") pod \"519b9db0-c559-4f08-be61-ea559d8a1189\" (UID: \"519b9db0-c559-4f08-be61-ea559d8a1189\") " Oct 30 13:10:45.959964 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:45.959854 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/519b9db0-c559-4f08-be61-ea559d8a1189-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/519b9db0-c559-4f08-be61-ea559d8a1189-maas-api-xks-ca\") pod \"519b9db0-c559-4f08-be61-ea559d8a1189\" (UID: \"519b9db0-c559-4f08-be61-ea559d8a1189\") " Oct 30 13:10:45.961423 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:45.961394 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/519b9db0-c559-4f08-be61-ea559d8a1189-maas-api-openshift-ca" pod "519b9db0-c559-4f08-be61-ea559d8a1189" (UID: "519b9db0-c559-4f08-be61-ea559d8a1189"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:45.963178 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:45.963148 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/519b9db0-c559-4f08-be61-ea559d8a1189-maas-api-xks-ca" pod "519b9db0-c559-4f08-be61-ea559d8a1189" (UID: "519b9db0-c559-4f08-be61-ea559d8a1189"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:10:45.963670 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:45.963640 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/519b9db0-c559-4f08-be61-ea559d8a1189-kube-api-access-q82q6" pod "519b9db0-c559-4f08-be61-ea559d8a1189" (UID: "519b9db0-c559-4f08-be61-ea559d8a1189"). InnerVolumeSpecName "kube-api-access-q82q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:46.062697 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:46.062626 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/519b9db0-c559-4f08-be61-ea559d8a1189-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:10:46.062697 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:46.062669 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q82q6\" (UniqueName: \"kubernetes.io/projected/519b9db0-c559-4f08-be61-ea559d8a1189-kube-api-access-q82q6\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:10:46.062697 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:46.062687 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/519b9db0-c559-4f08-be61-ea559d8a1189-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:10:46.303509 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:46.303416 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:10:46.303509 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:46.303476 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" podUID="0f2721af-325b-407d-8e1e-c1ed692291e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:46.703449 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:46.703399 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ec3f23-h7q9k" event={"ID":"519b9db0-c559-4f08-be61-ea559d8a1189","Type":"ContainerDied","Data":"5ad1fada40ceec17432933cdca340e62b7e5df120411f7c5ba33df99591cbe5d"} Oct 30 13:10:46.703662 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:46.703480 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ad1fada40ceec17432933cdca340e62b7e5df120411f7c5ba33df99591cbe5d" Oct 30 13:10:46.703662 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:46.703558 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ec3f23-h7q9k" Oct 30 13:10:47.534938 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.534902 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" Oct 30 13:10:47.614227 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.612880 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3680a3d2-0b13-40df-aeeb-6a924495f8d2-kube-api-access-z9srd\" (UniqueName: \"kubernetes.io/projected/3680a3d2-0b13-40df-aeeb-6a924495f8d2-kube-api-access-z9srd\") pod \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " Oct 30 13:10:47.614227 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.612990 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3680a3d2-0b13-40df-aeeb-6a924495f8d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3680a3d2-0b13-40df-aeeb-6a924495f8d2-workload-certs\") pod \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " Oct 30 13:10:47.614227 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.613026 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3680a3d2-0b13-40df-aeeb-6a924495f8d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3680a3d2-0b13-40df-aeeb-6a924495f8d2-workload-socket\") pod \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " Oct 30 13:10:47.614227 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.613056 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3680a3d2-0b13-40df-aeeb-6a924495f8d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3680a3d2-0b13-40df-aeeb-6a924495f8d2-istio-envoy\") pod \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " Oct 30 13:10:47.614227 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.613210 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3680a3d2-0b13-40df-aeeb-6a924495f8d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3680a3d2-0b13-40df-aeeb-6a924495f8d2-istio-data\") pod \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " Oct 30 13:10:47.614227 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.613254 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3680a3d2-0b13-40df-aeeb-6a924495f8d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3680a3d2-0b13-40df-aeeb-6a924495f8d2-istiod-ca-cert\") pod \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " Oct 30 13:10:47.614227 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.613373 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3680a3d2-0b13-40df-aeeb-6a924495f8d2-istio-token\" (UniqueName: \"kubernetes.io/projected/3680a3d2-0b13-40df-aeeb-6a924495f8d2-istio-token\") pod \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " Oct 30 13:10:47.614227 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.613428 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3680a3d2-0b13-40df-aeeb-6a924495f8d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3680a3d2-0b13-40df-aeeb-6a924495f8d2-istio-podinfo\") pod \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " Oct 30 13:10:47.614227 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.613490 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3680a3d2-0b13-40df-aeeb-6a924495f8d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3680a3d2-0b13-40df-aeeb-6a924495f8d2-credential-socket\") pod \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\" (UID: \"3680a3d2-0b13-40df-aeeb-6a924495f8d2\") " Oct 30 13:10:47.616704 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.616471 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3680a3d2-0b13-40df-aeeb-6a924495f8d2-credential-socket" pod "3680a3d2-0b13-40df-aeeb-6a924495f8d2" (UID: "3680a3d2-0b13-40df-aeeb-6a924495f8d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:10:47.619096 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.619065 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3680a3d2-0b13-40df-aeeb-6a924495f8d2-istio-data" pod "3680a3d2-0b13-40df-aeeb-6a924495f8d2" (UID: "3680a3d2-0b13-40df-aeeb-6a924495f8d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:10:47.619301 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.619153 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3680a3d2-0b13-40df-aeeb-6a924495f8d2-workload-socket" pod "3680a3d2-0b13-40df-aeeb-6a924495f8d2" (UID: "3680a3d2-0b13-40df-aeeb-6a924495f8d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:10:47.619472 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.619447 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3680a3d2-0b13-40df-aeeb-6a924495f8d2-workload-certs" pod "3680a3d2-0b13-40df-aeeb-6a924495f8d2" (UID: "3680a3d2-0b13-40df-aeeb-6a924495f8d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:10:47.619628 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.619606 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3680a3d2-0b13-40df-aeeb-6a924495f8d2-istiod-ca-cert" pod "3680a3d2-0b13-40df-aeeb-6a924495f8d2" (UID: "3680a3d2-0b13-40df-aeeb-6a924495f8d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:47.633664 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.633623 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3680a3d2-0b13-40df-aeeb-6a924495f8d2-istio-envoy" pod "3680a3d2-0b13-40df-aeeb-6a924495f8d2" (UID: "3680a3d2-0b13-40df-aeeb-6a924495f8d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:10:47.633664 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.633631 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3680a3d2-0b13-40df-aeeb-6a924495f8d2-istio-podinfo" pod "3680a3d2-0b13-40df-aeeb-6a924495f8d2" (UID: "3680a3d2-0b13-40df-aeeb-6a924495f8d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:10:47.634331 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.634231 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3680a3d2-0b13-40df-aeeb-6a924495f8d2-istio-token" pod "3680a3d2-0b13-40df-aeeb-6a924495f8d2" (UID: "3680a3d2-0b13-40df-aeeb-6a924495f8d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:47.634991 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.634965 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3680a3d2-0b13-40df-aeeb-6a924495f8d2-kube-api-access-z9srd" pod "3680a3d2-0b13-40df-aeeb-6a924495f8d2" (UID: "3680a3d2-0b13-40df-aeeb-6a924495f8d2"). InnerVolumeSpecName "kube-api-access-z9srd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:47.716699 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.716653 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3680a3d2-0b13-40df-aeeb-6a924495f8d2-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:10:47.716699 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.716701 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3680a3d2-0b13-40df-aeeb-6a924495f8d2-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:10:47.716990 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.716718 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3680a3d2-0b13-40df-aeeb-6a924495f8d2-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:10:47.716990 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.716733 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9srd\" (UniqueName: \"kubernetes.io/projected/3680a3d2-0b13-40df-aeeb-6a924495f8d2-kube-api-access-z9srd\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:10:47.716990 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.716747 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3680a3d2-0b13-40df-aeeb-6a924495f8d2-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:10:47.716990 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.716760 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3680a3d2-0b13-40df-aeeb-6a924495f8d2-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:10:47.716990 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.716775 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3680a3d2-0b13-40df-aeeb-6a924495f8d2-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:10:47.716990 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.716793 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3680a3d2-0b13-40df-aeeb-6a924495f8d2-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:10:47.716990 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.716807 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3680a3d2-0b13-40df-aeeb-6a924495f8d2-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:10:47.719566 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.719534 4931 generic.go:356] "Generic (PLEG): container finished" podID="3680a3d2-0b13-40df-aeeb-6a924495f8d2" containerID="9bce9341e990823fb376085bb78badbdae08554f3cfa22fa99036200228d18b0" exitCode=0 Oct 30 13:10:47.719721 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.719611 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" Oct 30 13:10:47.719791 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.719736 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" event={"ID":"3680a3d2-0b13-40df-aeeb-6a924495f8d2","Type":"ContainerDied","Data":"9bce9341e990823fb376085bb78badbdae08554f3cfa22fa99036200228d18b0"} Oct 30 13:10:47.719848 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.719795 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn" event={"ID":"3680a3d2-0b13-40df-aeeb-6a924495f8d2","Type":"ContainerDied","Data":"7643248a06a0e4ada7f90a17182efbf2ee07de735b207c89cb732bffbfbb96ab"} Oct 30 13:10:47.719848 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.719829 4931 scope.go:122] "RemoveContainer" containerID="9bce9341e990823fb376085bb78badbdae08554f3cfa22fa99036200228d18b0" Oct 30 13:10:47.818299 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.806353 4931 scope.go:122] "RemoveContainer" containerID="9bce9341e990823fb376085bb78badbdae08554f3cfa22fa99036200228d18b0" Oct 30 13:10:47.818299 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:10:47.815589 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bce9341e990823fb376085bb78badbdae08554f3cfa22fa99036200228d18b0\": container with ID starting with 9bce9341e990823fb376085bb78badbdae08554f3cfa22fa99036200228d18b0 not found: ID does not exist" containerID="9bce9341e990823fb376085bb78badbdae08554f3cfa22fa99036200228d18b0" Oct 30 13:10:47.818299 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.815638 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bce9341e990823fb376085bb78badbdae08554f3cfa22fa99036200228d18b0"} err="failed to get container status \"9bce9341e990823fb376085bb78badbdae08554f3cfa22fa99036200228d18b0\": rpc error: code = NotFound desc = could not find container \"9bce9341e990823fb376085bb78badbdae08554f3cfa22fa99036200228d18b0\": container with ID starting with 9bce9341e990823fb376085bb78badbdae08554f3cfa22fa99036200228d18b0 not found: ID does not exist" Oct 30 13:10:47.855683 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.846474 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn"] Oct 30 13:10:47.855683 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:47.852438 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn"] Oct 30 13:10:48.730897 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:48.730864 4931 generic.go:356] "Generic (PLEG): container finished" podID="0f2721af-325b-407d-8e1e-c1ed692291e8" containerID="ccf15ff02cc32911aaa6ab110724aee34fd6f8c45497292b7c656b93f277e9be" exitCode=0 Oct 30 13:10:48.731453 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:48.730935 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" event={"ID":"0f2721af-325b-407d-8e1e-c1ed692291e8","Type":"ContainerDied","Data":"ccf15ff02cc32911aaa6ab110724aee34fd6f8c45497292b7c656b93f277e9be"} Oct 30 13:10:48.731667 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:48.731648 4931 scope.go:122] "RemoveContainer" containerID="ccf15ff02cc32911aaa6ab110724aee34fd6f8c45497292b7c656b93f277e9be" Oct 30 13:10:48.732810 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:48.732791 4931 generic.go:356] "Generic (PLEG): container finished" podID="376b917f-91f0-4239-a1b3-eac12620d76e" containerID="f8058e44f1c9d82de9e43e6588266533dc232c20d660fc6cc6cf1dcd4178031c" exitCode=0 Oct 30 13:10:48.732873 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:48.732833 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" event={"ID":"376b917f-91f0-4239-a1b3-eac12620d76e","Type":"ContainerDied","Data":"f8058e44f1c9d82de9e43e6588266533dc232c20d660fc6cc6cf1dcd4178031c"} Oct 30 13:10:48.733399 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:48.733380 4931 scope.go:122] "RemoveContainer" containerID="f8058e44f1c9d82de9e43e6588266533dc232c20d660fc6cc6cf1dcd4178031c" Oct 30 13:10:48.733527 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:48.733501 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:48.733585 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:48.733572 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:48.733620 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:48.733605 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:48.734973 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:48.734945 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:48.735063 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:48.735023 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:48.735063 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:48.735057 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:49.515670 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:49.515624 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3680a3d2-0b13-40df-aeeb-6a924495f8d2" path="/var/lib/kubelet/pods/3680a3d2-0b13-40df-aeeb-6a924495f8d2/volumes" Oct 30 13:10:50.062297 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:50.057947 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" event={"ID":"0f2721af-325b-407d-8e1e-c1ed692291e8","Type":"ContainerStarted","Data":"21e09ea257786f910869b4a2138a4b0d8977f6fc51a882f77c64709d4224f27b"} Oct 30 13:10:50.062820 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:50.062391 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" event={"ID":"376b917f-91f0-4239-a1b3-eac12620d76e","Type":"ContainerStarted","Data":"03d514035c1e421392564bd70f0dcbed3806b16e4f6cf47d648d4c04a3b1df06"} Oct 30 13:10:50.902852 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:50.902813 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:10:50.903055 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:50.903031 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 13:10:50.903102 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:50.903084 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" podUID="0f2721af-325b-407d-8e1e-c1ed692291e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 13:10:51.004880 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:51.004841 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:10:51.005587 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:51.005547 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 13:10:51.005722 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:51.005615 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" podUID="376b917f-91f0-4239-a1b3-eac12620d76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 13:10:51.902779 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:51.902742 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 13:10:51.903186 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:51.902797 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" podUID="0f2721af-325b-407d-8e1e-c1ed692291e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 13:10:52.005028 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:52.004988 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 13:10:52.005236 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:52.005042 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" podUID="376b917f-91f0-4239-a1b3-eac12620d76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 13:10:52.902956 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:52.902910 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 13:10:52.903413 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:52.902988 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" podUID="0f2721af-325b-407d-8e1e-c1ed692291e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 13:10:53.005471 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:53.005433 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 13:10:53.005664 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:53.005487 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" podUID="376b917f-91f0-4239-a1b3-eac12620d76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 13:10:53.902752 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:53.902717 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 13:10:53.902948 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:53.902774 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" podUID="0f2721af-325b-407d-8e1e-c1ed692291e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 13:10:54.004699 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:54.004660 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 13:10:54.005110 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:54.004721 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" podUID="376b917f-91f0-4239-a1b3-eac12620d76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 13:10:54.902591 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:54.902543 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 13:10:54.902794 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:54.902635 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" podUID="0f2721af-325b-407d-8e1e-c1ed692291e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 13:10:55.005288 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:55.005234 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 13:10:55.005720 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:55.005319 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" podUID="376b917f-91f0-4239-a1b3-eac12620d76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 13:10:55.354883 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:55.354845 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:10:55.907306 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:55.907241 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:10:55.907567 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:55.907540 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:10:55.908221 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:55.908198 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:10:56.010494 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:56.010463 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:10:56.150978 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:56.150944 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn"] Oct 30 13:10:56.151222 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:56.151198 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" podUID="42c653e7-3421-4d55-b035-33d6f286413a" containerName="istio-proxy" containerID="cri-o://5b2cc422b1fedef338f30defec154e0f7289e8cc859731d96f7acfa5c82520b8" gracePeriod=30 Oct 30 13:10:56.221962 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:56.221935 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:10:58.484990 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:58.484956 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17fc433e-2kd6j"] Oct 30 13:10:58.485591 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:58.485577 4931 cpu_manager.go:414] "containerMap: removing container" podUID="519b9db0-c559-4f08-be61-ea559d8a1189" containerName="revoke-keys" Oct 30 13:10:58.485650 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:58.485592 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="519b9db0-c559-4f08-be61-ea559d8a1189" containerName="revoke-keys" Oct 30 13:10:58.485650 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:58.485619 4931 cpu_manager.go:414] "containerMap: removing container" podUID="3680a3d2-0b13-40df-aeeb-6a924495f8d2" containerName="istio-proxy" Oct 30 13:10:58.485650 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:58.485626 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3680a3d2-0b13-40df-aeeb-6a924495f8d2" containerName="istio-proxy" Oct 30 13:10:58.485760 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:58.485749 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="3680a3d2-0b13-40df-aeeb-6a924495f8d2" containerName="istio-proxy" Oct 30 13:10:58.485796 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:58.485768 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="519b9db0-c559-4f08-be61-ea559d8a1189" containerName="revoke-keys" Oct 30 13:10:58.489354 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:58.489338 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17fc433e-2kd6j" Oct 30 13:10:58.508587 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:58.508561 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17fc433e-2kd6j"] Oct 30 13:10:58.517241 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:58.517214 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:10:58.517434 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:58.517330 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:10:58.589293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:58.589227 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3cb3129-a627-402e-a401-46b48f4ec01a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-17fc433e-2kd6j\" (UID: \"b3cb3129-a627-402e-a401-46b48f4ec01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17fc433e-2kd6j" Oct 30 13:10:58.589506 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:58.589372 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wrnb\" (UniqueName: \"kubernetes.io/projected/b3cb3129-a627-402e-a401-46b48f4ec01a-kube-api-access-6wrnb\") pod \"maas-api-revoke-keys-e2e-mt-17fc433e-2kd6j\" (UID: \"b3cb3129-a627-402e-a401-46b48f4ec01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17fc433e-2kd6j" Oct 30 13:10:58.589650 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:58.589631 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3cb3129-a627-402e-a401-46b48f4ec01a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-17fc433e-2kd6j\" (UID: \"b3cb3129-a627-402e-a401-46b48f4ec01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17fc433e-2kd6j" Oct 30 13:10:58.691689 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:58.691656 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3cb3129-a627-402e-a401-46b48f4ec01a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-17fc433e-2kd6j\" (UID: \"b3cb3129-a627-402e-a401-46b48f4ec01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17fc433e-2kd6j" Oct 30 13:10:58.691879 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:58.691708 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wrnb\" (UniqueName: \"kubernetes.io/projected/b3cb3129-a627-402e-a401-46b48f4ec01a-kube-api-access-6wrnb\") pod \"maas-api-revoke-keys-e2e-mt-17fc433e-2kd6j\" (UID: \"b3cb3129-a627-402e-a401-46b48f4ec01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17fc433e-2kd6j" Oct 30 13:10:58.691956 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:58.691937 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3cb3129-a627-402e-a401-46b48f4ec01a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-17fc433e-2kd6j\" (UID: \"b3cb3129-a627-402e-a401-46b48f4ec01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17fc433e-2kd6j" Oct 30 13:10:58.692329 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:58.692305 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3cb3129-a627-402e-a401-46b48f4ec01a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-17fc433e-2kd6j\" (UID: \"b3cb3129-a627-402e-a401-46b48f4ec01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17fc433e-2kd6j" Oct 30 13:10:58.694321 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:58.694299 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3cb3129-a627-402e-a401-46b48f4ec01a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-17fc433e-2kd6j\" (UID: \"b3cb3129-a627-402e-a401-46b48f4ec01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17fc433e-2kd6j" Oct 30 13:10:58.723345 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:58.723316 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wrnb\" (UniqueName: \"kubernetes.io/projected/b3cb3129-a627-402e-a401-46b48f4ec01a-kube-api-access-6wrnb\") pod \"maas-api-revoke-keys-e2e-mt-17fc433e-2kd6j\" (UID: \"b3cb3129-a627-402e-a401-46b48f4ec01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17fc433e-2kd6j" Oct 30 13:10:58.803714 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:58.803636 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17fc433e-2kd6j" Oct 30 13:10:59.182902 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:59.182866 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17fc433e-2kd6j"] Oct 30 13:10:59.259344 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:10:59.259309 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17fc433e-2kd6j" event={"ID":"b3cb3129-a627-402e-a401-46b48f4ec01a","Type":"ContainerStarted","Data":"69da8118ce975cadbcdea280697f3e9b9ed0a791cc862219b26e70ced31d50d5"} Oct 30 13:11:00.268210 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:00.268177 4931 generic.go:356] "Generic (PLEG): container finished" podID="b3cb3129-a627-402e-a401-46b48f4ec01a" containerID="b930c236469dfbe1bc7260c6f159fbd238fd94a04552478c72afddc686e69da8" exitCode=0 Oct 30 13:11:00.268647 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:00.268262 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17fc433e-2kd6j" event={"ID":"b3cb3129-a627-402e-a401-46b48f4ec01a","Type":"ContainerDied","Data":"b930c236469dfbe1bc7260c6f159fbd238fd94a04552478c72afddc686e69da8"} Oct 30 13:11:01.277978 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.277940 4931 generic.go:356] "Generic (PLEG): container finished" podID="42c653e7-3421-4d55-b035-33d6f286413a" containerID="5b2cc422b1fedef338f30defec154e0f7289e8cc859731d96f7acfa5c82520b8" exitCode=0 Oct 30 13:11:01.278342 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.278007 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" event={"ID":"42c653e7-3421-4d55-b035-33d6f286413a","Type":"ContainerDied","Data":"5b2cc422b1fedef338f30defec154e0f7289e8cc859731d96f7acfa5c82520b8"} Oct 30 13:11:01.468408 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.468378 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" Oct 30 13:11:01.507291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.507197 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17fc433e-2kd6j" Oct 30 13:11:01.585194 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.585157 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42c653e7-3421-4d55-b035-33d6f286413a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42c653e7-3421-4d55-b035-33d6f286413a-istiod-ca-cert\") pod \"42c653e7-3421-4d55-b035-33d6f286413a\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " Oct 30 13:11:01.585524 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.585502 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3cb3129-a627-402e-a401-46b48f4ec01a-kube-api-access-6wrnb\" (UniqueName: \"kubernetes.io/projected/b3cb3129-a627-402e-a401-46b48f4ec01a-kube-api-access-6wrnb\") pod \"b3cb3129-a627-402e-a401-46b48f4ec01a\" (UID: \"b3cb3129-a627-402e-a401-46b48f4ec01a\") " Oct 30 13:11:01.585684 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.585667 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42c653e7-3421-4d55-b035-33d6f286413a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42c653e7-3421-4d55-b035-33d6f286413a-workload-certs\") pod \"42c653e7-3421-4d55-b035-33d6f286413a\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " Oct 30 13:11:01.585846 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.585833 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42c653e7-3421-4d55-b035-33d6f286413a-kube-api-access-tcphr\" (UniqueName: \"kubernetes.io/projected/42c653e7-3421-4d55-b035-33d6f286413a-kube-api-access-tcphr\") pod \"42c653e7-3421-4d55-b035-33d6f286413a\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " Oct 30 13:11:01.586097 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.586080 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42c653e7-3421-4d55-b035-33d6f286413a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c653e7-3421-4d55-b035-33d6f286413a-workload-socket\") pod \"42c653e7-3421-4d55-b035-33d6f286413a\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " Oct 30 13:11:01.586307 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.586291 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42c653e7-3421-4d55-b035-33d6f286413a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42c653e7-3421-4d55-b035-33d6f286413a-istio-data\") pod \"42c653e7-3421-4d55-b035-33d6f286413a\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " Oct 30 13:11:01.586425 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.586412 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42c653e7-3421-4d55-b035-33d6f286413a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42c653e7-3421-4d55-b035-33d6f286413a-istio-envoy\") pod \"42c653e7-3421-4d55-b035-33d6f286413a\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " Oct 30 13:11:01.586576 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.586563 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42c653e7-3421-4d55-b035-33d6f286413a-istio-token\" (UniqueName: \"kubernetes.io/projected/42c653e7-3421-4d55-b035-33d6f286413a-istio-token\") pod \"42c653e7-3421-4d55-b035-33d6f286413a\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " Oct 30 13:11:01.586703 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.586690 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3cb3129-a627-402e-a401-46b48f4ec01a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3cb3129-a627-402e-a401-46b48f4ec01a-maas-api-xks-ca\") pod \"b3cb3129-a627-402e-a401-46b48f4ec01a\" (UID: \"b3cb3129-a627-402e-a401-46b48f4ec01a\") " Oct 30 13:11:01.586814 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.586800 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3cb3129-a627-402e-a401-46b48f4ec01a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3cb3129-a627-402e-a401-46b48f4ec01a-maas-api-openshift-ca\") pod \"b3cb3129-a627-402e-a401-46b48f4ec01a\" (UID: \"b3cb3129-a627-402e-a401-46b48f4ec01a\") " Oct 30 13:11:01.586972 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.586958 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42c653e7-3421-4d55-b035-33d6f286413a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42c653e7-3421-4d55-b035-33d6f286413a-istio-podinfo\") pod \"42c653e7-3421-4d55-b035-33d6f286413a\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " Oct 30 13:11:01.587310 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.587290 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42c653e7-3421-4d55-b035-33d6f286413a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c653e7-3421-4d55-b035-33d6f286413a-credential-socket\") pod \"42c653e7-3421-4d55-b035-33d6f286413a\" (UID: \"42c653e7-3421-4d55-b035-33d6f286413a\") " Oct 30 13:11:01.593655 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.586459 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c653e7-3421-4d55-b035-33d6f286413a-workload-certs" pod "42c653e7-3421-4d55-b035-33d6f286413a" (UID: "42c653e7-3421-4d55-b035-33d6f286413a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:01.593655 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.586658 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c653e7-3421-4d55-b035-33d6f286413a-workload-socket" pod "42c653e7-3421-4d55-b035-33d6f286413a" (UID: "42c653e7-3421-4d55-b035-33d6f286413a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:01.593655 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.587251 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c653e7-3421-4d55-b035-33d6f286413a-istio-data" pod "42c653e7-3421-4d55-b035-33d6f286413a" (UID: "42c653e7-3421-4d55-b035-33d6f286413a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:01.593655 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.588447 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42c653e7-3421-4d55-b035-33d6f286413a-istiod-ca-cert" pod "42c653e7-3421-4d55-b035-33d6f286413a" (UID: "42c653e7-3421-4d55-b035-33d6f286413a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:01.593957 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.589094 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3cb3129-a627-402e-a401-46b48f4ec01a-maas-api-openshift-ca" pod "b3cb3129-a627-402e-a401-46b48f4ec01a" (UID: "b3cb3129-a627-402e-a401-46b48f4ec01a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:01.593957 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.593428 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3cb3129-a627-402e-a401-46b48f4ec01a-kube-api-access-6wrnb" pod "b3cb3129-a627-402e-a401-46b48f4ec01a" (UID: "b3cb3129-a627-402e-a401-46b48f4ec01a"). InnerVolumeSpecName "kube-api-access-6wrnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:01.593957 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.593621 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c653e7-3421-4d55-b035-33d6f286413a-credential-socket" pod "42c653e7-3421-4d55-b035-33d6f286413a" (UID: "42c653e7-3421-4d55-b035-33d6f286413a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:01.595860 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.595409 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42c653e7-3421-4d55-b035-33d6f286413a-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.595860 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.595434 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wrnb\" (UniqueName: \"kubernetes.io/projected/b3cb3129-a627-402e-a401-46b48f4ec01a-kube-api-access-6wrnb\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.595860 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.595447 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42c653e7-3421-4d55-b035-33d6f286413a-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.595860 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.595461 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c653e7-3421-4d55-b035-33d6f286413a-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.595860 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.595474 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42c653e7-3421-4d55-b035-33d6f286413a-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.595860 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.595487 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3cb3129-a627-402e-a401-46b48f4ec01a-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.595860 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.595498 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c653e7-3421-4d55-b035-33d6f286413a-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.597546 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.597507 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42c653e7-3421-4d55-b035-33d6f286413a-istio-podinfo" pod "42c653e7-3421-4d55-b035-33d6f286413a" (UID: "42c653e7-3421-4d55-b035-33d6f286413a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:11:01.600991 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.600950 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3cb3129-a627-402e-a401-46b48f4ec01a-maas-api-xks-ca" pod "b3cb3129-a627-402e-a401-46b48f4ec01a" (UID: "b3cb3129-a627-402e-a401-46b48f4ec01a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:11:01.601488 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.601467 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c653e7-3421-4d55-b035-33d6f286413a-istio-envoy" pod "42c653e7-3421-4d55-b035-33d6f286413a" (UID: "42c653e7-3421-4d55-b035-33d6f286413a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:01.603312 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.602576 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42c653e7-3421-4d55-b035-33d6f286413a-istio-token" pod "42c653e7-3421-4d55-b035-33d6f286413a" (UID: "42c653e7-3421-4d55-b035-33d6f286413a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:01.611508 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.611395 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42c653e7-3421-4d55-b035-33d6f286413a-kube-api-access-tcphr" pod "42c653e7-3421-4d55-b035-33d6f286413a" (UID: "42c653e7-3421-4d55-b035-33d6f286413a"). InnerVolumeSpecName "kube-api-access-tcphr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:01.696556 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.696523 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42c653e7-3421-4d55-b035-33d6f286413a-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.696556 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.696555 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3cb3129-a627-402e-a401-46b48f4ec01a-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.696772 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.696568 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42c653e7-3421-4d55-b035-33d6f286413a-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.696772 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.696580 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcphr\" (UniqueName: \"kubernetes.io/projected/42c653e7-3421-4d55-b035-33d6f286413a-kube-api-access-tcphr\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.696772 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.696593 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42c653e7-3421-4d55-b035-33d6f286413a-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.979380 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.979301 4931 scope.go:122] "RemoveContainer" containerID="bb77f2c13c6969a6965ecfac0052da2a23bb3170f06e2406ddec59ce3985762f" Oct 30 13:11:01.990057 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.990035 4931 scope.go:122] "RemoveContainer" containerID="a8aaf71ebc00a815442a4c3952434f814b30d77eb80accd3cfb10b6ad4ecf86b" Oct 30 13:11:01.999831 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:01.999807 4931 scope.go:122] "RemoveContainer" containerID="87bdb23de0a1d179688d66eb1f0a3c1c8af90b11744366b4e31d8bc9b8e0cb27" Oct 30 13:11:02.009141 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:02.009120 4931 scope.go:122] "RemoveContainer" containerID="72d88217d611297879e67ee0c32880568bb27066f7ccb266a6447802910def8c" Oct 30 13:11:02.018577 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:02.018555 4931 scope.go:122] "RemoveContainer" containerID="b995d3ad578c5561c25b57890b0bea80c1f1ef95748500f6376751faa25e6274" Oct 30 13:11:02.027783 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:02.027764 4931 scope.go:122] "RemoveContainer" containerID="5b2cc422b1fedef338f30defec154e0f7289e8cc859731d96f7acfa5c82520b8" Oct 30 13:11:02.037161 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:02.037141 4931 scope.go:122] "RemoveContainer" containerID="60ee994d1b964ffda9de6f562cf2c97bba845388a1188f66c973fd478d4319f8" Oct 30 13:11:02.286000 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:02.285891 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" event={"ID":"42c653e7-3421-4d55-b035-33d6f286413a","Type":"ContainerDied","Data":"650e4f21a82e7439efdc7c1ae32bbf65f4b1a698522b8c1e78228cda28503de8"} Oct 30 13:11:02.287668 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:02.287648 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17fc433e-2kd6j" Oct 30 13:11:02.287746 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:02.287680 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17fc433e-2kd6j" event={"ID":"b3cb3129-a627-402e-a401-46b48f4ec01a","Type":"ContainerDied","Data":"69da8118ce975cadbcdea280697f3e9b9ed0a791cc862219b26e70ced31d50d5"} Oct 30 13:11:02.287746 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:02.287726 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69da8118ce975cadbcdea280697f3e9b9ed0a791cc862219b26e70ced31d50d5" Oct 30 13:11:02.287814 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:02.287795 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn" Oct 30 13:11:02.408324 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:02.408290 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn"] Oct 30 13:11:02.415249 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:02.415209 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn"] Oct 30 13:11:03.517663 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:03.517618 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42c653e7-3421-4d55-b035-33d6f286413a" path="/var/lib/kubelet/pods/42c653e7-3421-4d55-b035-33d6f286413a/volumes" Oct 30 13:11:05.188751 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:05.188711 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:11:07.378634 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:11:07.378593 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:11:19.168413 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:19.168182 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17fc433e-2kd6j"] Oct 30 13:11:19.194208 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:19.194170 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17fc433e-2kd6j"] Oct 30 13:11:19.514888 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:19.514854 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3cb3129-a627-402e-a401-46b48f4ec01a" path="/var/lib/kubelet/pods/b3cb3129-a627-402e-a401-46b48f4ec01a/volumes" Oct 30 13:11:21.136839 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:21.136805 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7"] Oct 30 13:11:21.137643 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:21.137620 4931 cpu_manager.go:414] "containerMap: removing container" podUID="42c653e7-3421-4d55-b035-33d6f286413a" containerName="istio-proxy" Oct 30 13:11:21.137763 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:21.137644 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42c653e7-3421-4d55-b035-33d6f286413a" containerName="istio-proxy" Oct 30 13:11:21.137763 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:21.137701 4931 cpu_manager.go:414] "containerMap: removing container" podUID="b3cb3129-a627-402e-a401-46b48f4ec01a" containerName="revoke-keys" Oct 30 13:11:21.137763 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:21.137710 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3cb3129-a627-402e-a401-46b48f4ec01a" containerName="revoke-keys" Oct 30 13:11:21.137929 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:21.137882 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3cb3129-a627-402e-a401-46b48f4ec01a" containerName="revoke-keys" Oct 30 13:11:21.137929 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:21.137914 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="42c653e7-3421-4d55-b035-33d6f286413a" containerName="istio-proxy" Oct 30 13:11:21.143721 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:21.143695 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7" Oct 30 13:11:21.164507 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:21.164472 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7"] Oct 30 13:11:21.173845 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:21.173798 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-31e10b-serving-cert\"" Oct 30 13:11:21.219227 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:21.219190 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6acaa883-abd5-40b0-b62d-2aaf38ad32ea-metrics-certs\") pod \"maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7\" (UID: \"6acaa883-abd5-40b0-b62d-2aaf38ad32ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7" Oct 30 13:11:21.219467 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:21.219306 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6acaa883-abd5-40b0-b62d-2aaf38ad32ea-openshift-service-ca\") pod \"maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7\" (UID: \"6acaa883-abd5-40b0-b62d-2aaf38ad32ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7" Oct 30 13:11:21.219516 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:21.219464 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6acaa883-abd5-40b0-b62d-2aaf38ad32ea-maas-api-tls\") pod \"maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7\" (UID: \"6acaa883-abd5-40b0-b62d-2aaf38ad32ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7" Oct 30 13:11:21.219573 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:21.219515 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgp6x\" (UniqueName: \"kubernetes.io/projected/6acaa883-abd5-40b0-b62d-2aaf38ad32ea-kube-api-access-vgp6x\") pod \"maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7\" (UID: \"6acaa883-abd5-40b0-b62d-2aaf38ad32ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7" Oct 30 13:11:21.321667 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:21.321628 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6acaa883-abd5-40b0-b62d-2aaf38ad32ea-metrics-certs\") pod \"maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7\" (UID: \"6acaa883-abd5-40b0-b62d-2aaf38ad32ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7" Oct 30 13:11:21.321842 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:21.321753 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6acaa883-abd5-40b0-b62d-2aaf38ad32ea-openshift-service-ca\") pod \"maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7\" (UID: \"6acaa883-abd5-40b0-b62d-2aaf38ad32ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7" Oct 30 13:11:21.321936 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:21.321874 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6acaa883-abd5-40b0-b62d-2aaf38ad32ea-maas-api-tls\") pod \"maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7\" (UID: \"6acaa883-abd5-40b0-b62d-2aaf38ad32ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7" Oct 30 13:11:21.321999 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:21.321944 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgp6x\" (UniqueName: \"kubernetes.io/projected/6acaa883-abd5-40b0-b62d-2aaf38ad32ea-kube-api-access-vgp6x\") pod \"maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7\" (UID: \"6acaa883-abd5-40b0-b62d-2aaf38ad32ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7" Oct 30 13:11:21.322716 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:21.322690 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6acaa883-abd5-40b0-b62d-2aaf38ad32ea-openshift-service-ca\") pod \"maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7\" (UID: \"6acaa883-abd5-40b0-b62d-2aaf38ad32ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7" Oct 30 13:11:21.324434 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:21.324408 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6acaa883-abd5-40b0-b62d-2aaf38ad32ea-maas-api-tls\") pod \"maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7\" (UID: \"6acaa883-abd5-40b0-b62d-2aaf38ad32ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7" Oct 30 13:11:21.324528 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:21.324459 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6acaa883-abd5-40b0-b62d-2aaf38ad32ea-metrics-certs\") pod \"maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7\" (UID: \"6acaa883-abd5-40b0-b62d-2aaf38ad32ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7" Oct 30 13:11:21.347313 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:21.347281 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgp6x\" (UniqueName: \"kubernetes.io/projected/6acaa883-abd5-40b0-b62d-2aaf38ad32ea-kube-api-access-vgp6x\") pod \"maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7\" (UID: \"6acaa883-abd5-40b0-b62d-2aaf38ad32ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7" Oct 30 13:11:21.459610 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:21.459569 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7" Oct 30 13:11:21.813519 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:21.813475 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7"] Oct 30 13:11:21.813695 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:11:21.813667 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6acaa883_abd5_40b0_b62d_2aaf38ad32ea.slice/crio-cd912d75dddc1d132dc5b93a749b508ac0dc2bc87b59001a5681897be1870561 WatchSource:0}: Error finding container cd912d75dddc1d132dc5b93a749b508ac0dc2bc87b59001a5681897be1870561: Status 404 returned error can't find the container with id cd912d75dddc1d132dc5b93a749b508ac0dc2bc87b59001a5681897be1870561 Oct 30 13:11:22.782649 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:22.782604 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7" event={"ID":"6acaa883-abd5-40b0-b62d-2aaf38ad32ea","Type":"ContainerStarted","Data":"6b11d1d7e7819dc83615ab3b1017e3eace3ab7c64569cf5919e1278c388fa2b7"} Oct 30 13:11:22.783068 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:22.782675 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7" Oct 30 13:11:22.783068 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:22.782700 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7" event={"ID":"6acaa883-abd5-40b0-b62d-2aaf38ad32ea","Type":"ContainerStarted","Data":"cd912d75dddc1d132dc5b93a749b508ac0dc2bc87b59001a5681897be1870561"} Oct 30 13:11:22.844290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:22.838576 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7" podStartSLOduration=1.466507222 podStartE2EDuration="1.83855695s" totalImagesPullingTime="372.049728ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:22.835350114 +0000 UTC m=+1581.438848583" watchObservedRunningTime="2026-10-30 13:11:22.83855695 +0000 UTC m=+1581.442055398" Oct 30 13:11:28.800082 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:28.800051 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7" Oct 30 13:11:33.053646 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:33.053603 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02b7cd66-7587p"] Oct 30 13:11:33.057812 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:33.057783 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02b7cd66-7587p"] Oct 30 13:11:33.514632 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:33.514599 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d706c855-11e7-443a-9dbf-00b9577f5b9e" path="/var/lib/kubelet/pods/d706c855-11e7-443a-9dbf-00b9577f5b9e/volumes" Oct 30 13:11:36.301492 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.301458 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q"] Oct 30 13:11:36.307161 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.307138 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" Oct 30 13:11:36.342524 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.342490 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-1c0e6c42-openshift-default-dockercfg-zstsq\"" Oct 30 13:11:36.344205 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.344169 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q"] Oct 30 13:11:36.344683 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.344654 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-istio-token\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" Oct 30 13:11:36.344780 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.344745 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-istio-data\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" Oct 30 13:11:36.344852 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.344794 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-istio-envoy\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" Oct 30 13:11:36.344852 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.344843 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-workload-socket\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" Oct 30 13:11:36.344990 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.344972 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pr5t\" (UniqueName: \"kubernetes.io/projected/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-kube-api-access-7pr5t\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" Oct 30 13:11:36.345140 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.345116 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-credential-socket\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" Oct 30 13:11:36.345291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.345166 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-workload-certs\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" Oct 30 13:11:36.345291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.345197 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-istiod-ca-cert\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" Oct 30 13:11:36.345291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.345245 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-istio-podinfo\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" Oct 30 13:11:36.506107 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.506058 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pr5t\" (UniqueName: \"kubernetes.io/projected/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-kube-api-access-7pr5t\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" Oct 30 13:11:36.506365 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.506242 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-credential-socket\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" Oct 30 13:11:36.506365 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.506319 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-workload-certs\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" Oct 30 13:11:36.506365 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.506357 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-istiod-ca-cert\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" Oct 30 13:11:36.506555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.506390 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-istio-podinfo\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" Oct 30 13:11:36.506555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.506500 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-istio-token\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" Oct 30 13:11:36.506663 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.506593 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-istio-data\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" Oct 30 13:11:36.506663 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.506648 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-istio-envoy\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" Oct 30 13:11:36.506764 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.506707 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-workload-socket\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" Oct 30 13:11:36.509218 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.509182 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-workload-socket\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" Oct 30 13:11:36.509847 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.509822 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-credential-socket\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" Oct 30 13:11:36.510061 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.510044 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-workload-certs\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" Oct 30 13:11:36.510563 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.510543 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-istiod-ca-cert\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" Oct 30 13:11:36.512709 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.512679 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-istio-data\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" Oct 30 13:11:36.513203 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.513169 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-istio-podinfo\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" Oct 30 13:11:36.516661 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.516634 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-istio-envoy\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" Oct 30 13:11:36.544602 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.544562 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pr5t\" (UniqueName: \"kubernetes.io/projected/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-kube-api-access-7pr5t\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" Oct 30 13:11:36.548003 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.547972 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-istio-token\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" Oct 30 13:11:36.623752 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.623654 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" Oct 30 13:11:36.964877 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:11:36.964847 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a13ccd_7ec5_4401_b316_77e26b4c6f7b.slice/crio-7ddcab40e3e84bc1efde1ac2887a918a05f196404f11e0af83463f688f7b13a0 WatchSource:0}: Error finding container 7ddcab40e3e84bc1efde1ac2887a918a05f196404f11e0af83463f688f7b13a0: Status 404 returned error can't find the container with id 7ddcab40e3e84bc1efde1ac2887a918a05f196404f11e0af83463f688f7b13a0 Oct 30 13:11:36.965171 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.965148 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q"] Oct 30 13:11:36.967879 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.967847 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:11:36.967962 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.967915 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:11:36.967962 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:36.967956 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:11:37.354327 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:11:37.354224 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:11:37.952302 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:37.951314 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" event={"ID":"20a13ccd-7ec5-4401-b316-77e26b4c6f7b","Type":"ContainerStarted","Data":"7ad486931063396f67048d748c0088b6fb03845f0e8b3f3416573523a613475b"} Oct 30 13:11:37.952302 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:37.951423 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" event={"ID":"20a13ccd-7ec5-4401-b316-77e26b4c6f7b","Type":"ContainerStarted","Data":"7ddcab40e3e84bc1efde1ac2887a918a05f196404f11e0af83463f688f7b13a0"} Oct 30 13:11:38.031315 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:38.031242 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" podStartSLOduration=2.031225506 podStartE2EDuration="2.031225506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:38.013869518 +0000 UTC m=+1596.617367965" watchObservedRunningTime="2026-10-30 13:11:38.031225506 +0000 UTC m=+1596.634723952" Oct 30 13:11:38.623893 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:38.623857 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" Oct 30 13:11:38.625058 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:38.625033 4931 patch_prober.go:29] interesting pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 13:11:38.625183 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:38.625088 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" podUID="20a13ccd-7ec5-4401-b316-77e26b4c6f7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 13:11:39.626238 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:39.626171 4931 patch_prober.go:29] interesting pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 13:11:39.626238 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:39.626239 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" podUID="20a13ccd-7ec5-4401-b316-77e26b4c6f7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 13:11:40.624722 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:40.624681 4931 patch_prober.go:29] interesting pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 13:11:40.624912 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:40.624747 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" podUID="20a13ccd-7ec5-4401-b316-77e26b4c6f7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 13:11:41.624363 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:41.624325 4931 patch_prober.go:29] interesting pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 13:11:41.624779 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:41.624383 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" podUID="20a13ccd-7ec5-4401-b316-77e26b4c6f7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 13:11:42.624845 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:42.624805 4931 patch_prober.go:29] interesting pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 13:11:42.625397 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:42.624869 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" podUID="20a13ccd-7ec5-4401-b316-77e26b4c6f7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 13:11:43.058294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:43.057322 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-015dc049-clz92"] Oct 30 13:11:43.066144 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:43.066106 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-015dc049-clz92"] Oct 30 13:11:43.515194 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:43.515160 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ca8cd79-0cc2-4813-aa69-e656ab2a1718" path="/var/lib/kubelet/pods/3ca8cd79-0cc2-4813-aa69-e656ab2a1718/volumes" Oct 30 13:11:43.641293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:43.624651 4931 patch_prober.go:29] interesting pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 13:11:43.641293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:43.624711 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" podUID="20a13ccd-7ec5-4401-b316-77e26b4c6f7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 13:11:44.661768 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:44.661735 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" Oct 30 13:11:44.662390 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:44.662253 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" Oct 30 13:11:44.663665 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:44.663636 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" Oct 30 13:11:45.753800 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:11:45.753762 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:12:02.120103 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:02.120070 4931 scope.go:122] "RemoveContainer" containerID="a8a97a8ab1dad74142ecc6acd13bb7f10e22c503f0bc43f36589afde4c018c79" Oct 30 13:12:02.131566 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:02.131540 4931 scope.go:122] "RemoveContainer" containerID="f1362f87bb51958f3c25195c62dd62a1d1e9c36969afa002ba31d7b2c50151dd" Oct 30 13:12:04.821335 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:04.821293 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58f9b7b5-xsk46"] Oct 30 13:12:04.827389 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:04.827363 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f9b7b5-xsk46" Oct 30 13:12:04.845734 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:04.845698 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58f9b7b5-xsk46"] Oct 30 13:12:04.853430 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:04.853386 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:12:04.853610 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:04.853515 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:12:04.920926 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:04.920892 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwfmk\" (UniqueName: \"kubernetes.io/projected/d85da789-4fa7-41e5-a7f3-6b16031020fe-kube-api-access-pwfmk\") pod \"maas-api-revoke-sub-58f9b7b5-xsk46\" (UID: \"d85da789-4fa7-41e5-a7f3-6b16031020fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f9b7b5-xsk46" Oct 30 13:12:04.921118 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:04.920961 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85da789-4fa7-41e5-a7f3-6b16031020fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58f9b7b5-xsk46\" (UID: \"d85da789-4fa7-41e5-a7f3-6b16031020fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f9b7b5-xsk46" Oct 30 13:12:04.921164 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:04.921121 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85da789-4fa7-41e5-a7f3-6b16031020fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58f9b7b5-xsk46\" (UID: \"d85da789-4fa7-41e5-a7f3-6b16031020fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f9b7b5-xsk46" Oct 30 13:12:05.022551 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:05.022498 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwfmk\" (UniqueName: \"kubernetes.io/projected/d85da789-4fa7-41e5-a7f3-6b16031020fe-kube-api-access-pwfmk\") pod \"maas-api-revoke-sub-58f9b7b5-xsk46\" (UID: \"d85da789-4fa7-41e5-a7f3-6b16031020fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f9b7b5-xsk46" Oct 30 13:12:05.022775 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:05.022612 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85da789-4fa7-41e5-a7f3-6b16031020fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58f9b7b5-xsk46\" (UID: \"d85da789-4fa7-41e5-a7f3-6b16031020fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f9b7b5-xsk46" Oct 30 13:12:05.022775 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:05.022728 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85da789-4fa7-41e5-a7f3-6b16031020fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58f9b7b5-xsk46\" (UID: \"d85da789-4fa7-41e5-a7f3-6b16031020fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f9b7b5-xsk46" Oct 30 13:12:05.023373 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:05.023347 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85da789-4fa7-41e5-a7f3-6b16031020fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58f9b7b5-xsk46\" (UID: \"d85da789-4fa7-41e5-a7f3-6b16031020fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f9b7b5-xsk46" Oct 30 13:12:05.025062 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:05.025033 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85da789-4fa7-41e5-a7f3-6b16031020fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58f9b7b5-xsk46\" (UID: \"d85da789-4fa7-41e5-a7f3-6b16031020fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f9b7b5-xsk46" Oct 30 13:12:05.047701 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:05.047669 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwfmk\" (UniqueName: \"kubernetes.io/projected/d85da789-4fa7-41e5-a7f3-6b16031020fe-kube-api-access-pwfmk\") pod \"maas-api-revoke-sub-58f9b7b5-xsk46\" (UID: \"d85da789-4fa7-41e5-a7f3-6b16031020fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f9b7b5-xsk46" Oct 30 13:12:05.143147 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:05.143055 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f9b7b5-xsk46" Oct 30 13:12:05.454364 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:12:05.454332 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd85da789_4fa7_41e5_a7f3_6b16031020fe.slice/crio-ff6a1908f8815ef005f8b35636f9cfad665f1a4b68672e85754b9e3a7bc1f741 WatchSource:0}: Error finding container ff6a1908f8815ef005f8b35636f9cfad665f1a4b68672e85754b9e3a7bc1f741: Status 404 returned error can't find the container with id ff6a1908f8815ef005f8b35636f9cfad665f1a4b68672e85754b9e3a7bc1f741 Oct 30 13:12:05.454991 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:05.454969 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58f9b7b5-xsk46"] Oct 30 13:12:06.348204 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:06.348166 4931 generic.go:356] "Generic (PLEG): container finished" podID="d85da789-4fa7-41e5-a7f3-6b16031020fe" containerID="b127bd0664b664dd8e588f6157e40f2e32cc940aaa6bad77c5bc22d781be44a5" exitCode=0 Oct 30 13:12:06.348204 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:06.348219 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f9b7b5-xsk46" event={"ID":"d85da789-4fa7-41e5-a7f3-6b16031020fe","Type":"ContainerDied","Data":"b127bd0664b664dd8e588f6157e40f2e32cc940aaa6bad77c5bc22d781be44a5"} Oct 30 13:12:06.348204 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:06.348260 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f9b7b5-xsk46" event={"ID":"d85da789-4fa7-41e5-a7f3-6b16031020fe","Type":"ContainerStarted","Data":"ff6a1908f8815ef005f8b35636f9cfad665f1a4b68672e85754b9e3a7bc1f741"} Oct 30 13:12:07.390325 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:12:07.390233 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:12:07.542150 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:07.542121 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f9b7b5-xsk46" Oct 30 13:12:07.674455 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:07.674356 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d85da789-4fa7-41e5-a7f3-6b16031020fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85da789-4fa7-41e5-a7f3-6b16031020fe-maas-api-xks-ca\") pod \"d85da789-4fa7-41e5-a7f3-6b16031020fe\" (UID: \"d85da789-4fa7-41e5-a7f3-6b16031020fe\") " Oct 30 13:12:07.675649 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:07.675620 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d85da789-4fa7-41e5-a7f3-6b16031020fe-kube-api-access-pwfmk\" (UniqueName: \"kubernetes.io/projected/d85da789-4fa7-41e5-a7f3-6b16031020fe-kube-api-access-pwfmk\") pod \"d85da789-4fa7-41e5-a7f3-6b16031020fe\" (UID: \"d85da789-4fa7-41e5-a7f3-6b16031020fe\") " Oct 30 13:12:07.675807 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:07.675769 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d85da789-4fa7-41e5-a7f3-6b16031020fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85da789-4fa7-41e5-a7f3-6b16031020fe-maas-api-openshift-ca\") pod \"d85da789-4fa7-41e5-a7f3-6b16031020fe\" (UID: \"d85da789-4fa7-41e5-a7f3-6b16031020fe\") " Oct 30 13:12:07.676285 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:07.676240 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d85da789-4fa7-41e5-a7f3-6b16031020fe-maas-api-openshift-ca" pod "d85da789-4fa7-41e5-a7f3-6b16031020fe" (UID: "d85da789-4fa7-41e5-a7f3-6b16031020fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:07.676621 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:07.676596 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85da789-4fa7-41e5-a7f3-6b16031020fe-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:12:07.679648 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:07.679621 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d85da789-4fa7-41e5-a7f3-6b16031020fe-maas-api-xks-ca" pod "d85da789-4fa7-41e5-a7f3-6b16031020fe" (UID: "d85da789-4fa7-41e5-a7f3-6b16031020fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:12:07.679782 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:07.679764 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d85da789-4fa7-41e5-a7f3-6b16031020fe-kube-api-access-pwfmk" pod "d85da789-4fa7-41e5-a7f3-6b16031020fe" (UID: "d85da789-4fa7-41e5-a7f3-6b16031020fe"). InnerVolumeSpecName "kube-api-access-pwfmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:07.783291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:07.782827 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85da789-4fa7-41e5-a7f3-6b16031020fe-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:12:07.783291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:07.782864 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwfmk\" (UniqueName: \"kubernetes.io/projected/d85da789-4fa7-41e5-a7f3-6b16031020fe-kube-api-access-pwfmk\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:12:08.397012 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:08.396944 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f9b7b5-xsk46" event={"ID":"d85da789-4fa7-41e5-a7f3-6b16031020fe","Type":"ContainerDied","Data":"ff6a1908f8815ef005f8b35636f9cfad665f1a4b68672e85754b9e3a7bc1f741"} Oct 30 13:12:08.397566 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:08.397034 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff6a1908f8815ef005f8b35636f9cfad665f1a4b68672e85754b9e3a7bc1f741" Oct 30 13:12:08.397566 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:08.397105 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f9b7b5-xsk46" Oct 30 13:12:12.158526 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:12.158487 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75abce-66k7k"] Oct 30 13:12:12.159540 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:12.159518 4931 cpu_manager.go:414] "containerMap: removing container" podUID="d85da789-4fa7-41e5-a7f3-6b16031020fe" containerName="revoke-keys" Oct 30 13:12:12.159615 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:12.159543 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d85da789-4fa7-41e5-a7f3-6b16031020fe" containerName="revoke-keys" Oct 30 13:12:12.159726 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:12.159713 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="d85da789-4fa7-41e5-a7f3-6b16031020fe" containerName="revoke-keys" Oct 30 13:12:12.164247 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:12.164223 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75abce-66k7k" Oct 30 13:12:12.183865 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:12.183832 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75abce-66k7k"] Oct 30 13:12:12.188644 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:12.188617 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:12:12.188832 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:12.188809 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:12:12.264177 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:12.264133 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a566f0ac-617b-4f46-a46d-faaae0391a93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-75abce-66k7k\" (UID: \"a566f0ac-617b-4f46-a46d-faaae0391a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75abce-66k7k" Oct 30 13:12:12.264377 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:12.264199 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4pmq\" (UniqueName: \"kubernetes.io/projected/a566f0ac-617b-4f46-a46d-faaae0391a93-kube-api-access-k4pmq\") pod \"maas-api-revoke-keys-e2e-api-a-75abce-66k7k\" (UID: \"a566f0ac-617b-4f46-a46d-faaae0391a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75abce-66k7k" Oct 30 13:12:12.264469 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:12.264445 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a566f0ac-617b-4f46-a46d-faaae0391a93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-75abce-66k7k\" (UID: \"a566f0ac-617b-4f46-a46d-faaae0391a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75abce-66k7k" Oct 30 13:12:12.365474 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:12.365438 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a566f0ac-617b-4f46-a46d-faaae0391a93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-75abce-66k7k\" (UID: \"a566f0ac-617b-4f46-a46d-faaae0391a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75abce-66k7k" Oct 30 13:12:12.365683 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:12.365490 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4pmq\" (UniqueName: \"kubernetes.io/projected/a566f0ac-617b-4f46-a46d-faaae0391a93-kube-api-access-k4pmq\") pod \"maas-api-revoke-keys-e2e-api-a-75abce-66k7k\" (UID: \"a566f0ac-617b-4f46-a46d-faaae0391a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75abce-66k7k" Oct 30 13:12:12.365683 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:12.365589 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a566f0ac-617b-4f46-a46d-faaae0391a93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-75abce-66k7k\" (UID: \"a566f0ac-617b-4f46-a46d-faaae0391a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75abce-66k7k" Oct 30 13:12:12.366185 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:12.366157 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a566f0ac-617b-4f46-a46d-faaae0391a93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-75abce-66k7k\" (UID: \"a566f0ac-617b-4f46-a46d-faaae0391a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75abce-66k7k" Oct 30 13:12:12.367892 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:12.367866 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a566f0ac-617b-4f46-a46d-faaae0391a93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-75abce-66k7k\" (UID: \"a566f0ac-617b-4f46-a46d-faaae0391a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75abce-66k7k" Oct 30 13:12:12.390042 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:12.390018 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4pmq\" (UniqueName: \"kubernetes.io/projected/a566f0ac-617b-4f46-a46d-faaae0391a93-kube-api-access-k4pmq\") pod \"maas-api-revoke-keys-e2e-api-a-75abce-66k7k\" (UID: \"a566f0ac-617b-4f46-a46d-faaae0391a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75abce-66k7k" Oct 30 13:12:12.482601 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:12.482569 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75abce-66k7k" Oct 30 13:12:12.789181 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:12.789146 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75abce-66k7k"] Oct 30 13:12:12.789669 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:12:12.789644 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda566f0ac_617b_4f46_a46d_faaae0391a93.slice/crio-a8f9c445c1141bcce57c7bc071b54d42d97e35b388f0531839113821707d33f1 WatchSource:0}: Error finding container a8f9c445c1141bcce57c7bc071b54d42d97e35b388f0531839113821707d33f1: Status 404 returned error can't find the container with id a8f9c445c1141bcce57c7bc071b54d42d97e35b388f0531839113821707d33f1 Oct 30 13:12:13.447766 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:13.447726 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75abce-66k7k" event={"ID":"a566f0ac-617b-4f46-a46d-faaae0391a93","Type":"ContainerStarted","Data":"25a1941ee4642a72828a9cd2b8c95a011309390c8b7719a05c8a451fb485eb3f"} Oct 30 13:12:13.448213 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:13.447799 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75abce-66k7k" event={"ID":"a566f0ac-617b-4f46-a46d-faaae0391a93","Type":"ContainerStarted","Data":"a8f9c445c1141bcce57c7bc071b54d42d97e35b388f0531839113821707d33f1"} Oct 30 13:12:13.494810 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:13.494760 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75abce-66k7k" podStartSLOduration=1.494744939 podStartE2EDuration="1.494744939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:13.492838859 +0000 UTC m=+1632.096337335" watchObservedRunningTime="2026-10-30 13:12:13.494744939 +0000 UTC m=+1632.098243384" Oct 30 13:12:14.460197 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:14.460149 4931 generic.go:356] "Generic (PLEG): container finished" podID="a566f0ac-617b-4f46-a46d-faaae0391a93" containerID="25a1941ee4642a72828a9cd2b8c95a011309390c8b7719a05c8a451fb485eb3f" exitCode=0 Oct 30 13:12:14.460730 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:14.460470 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75abce-66k7k" event={"ID":"a566f0ac-617b-4f46-a46d-faaae0391a93","Type":"ContainerDied","Data":"25a1941ee4642a72828a9cd2b8c95a011309390c8b7719a05c8a451fb485eb3f"} Oct 30 13:12:15.635817 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:15.635784 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75abce-66k7k" Oct 30 13:12:15.720432 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:15.720326 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a566f0ac-617b-4f46-a46d-faaae0391a93-kube-api-access-k4pmq\" (UniqueName: \"kubernetes.io/projected/a566f0ac-617b-4f46-a46d-faaae0391a93-kube-api-access-k4pmq\") pod \"a566f0ac-617b-4f46-a46d-faaae0391a93\" (UID: \"a566f0ac-617b-4f46-a46d-faaae0391a93\") " Oct 30 13:12:15.720622 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:15.720431 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a566f0ac-617b-4f46-a46d-faaae0391a93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a566f0ac-617b-4f46-a46d-faaae0391a93-maas-api-openshift-ca\") pod \"a566f0ac-617b-4f46-a46d-faaae0391a93\" (UID: \"a566f0ac-617b-4f46-a46d-faaae0391a93\") " Oct 30 13:12:15.720622 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:15.720497 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a566f0ac-617b-4f46-a46d-faaae0391a93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a566f0ac-617b-4f46-a46d-faaae0391a93-maas-api-xks-ca\") pod \"a566f0ac-617b-4f46-a46d-faaae0391a93\" (UID: \"a566f0ac-617b-4f46-a46d-faaae0391a93\") " Oct 30 13:12:15.720835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:15.720802 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a566f0ac-617b-4f46-a46d-faaae0391a93-maas-api-openshift-ca" pod "a566f0ac-617b-4f46-a46d-faaae0391a93" (UID: "a566f0ac-617b-4f46-a46d-faaae0391a93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:15.721425 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:15.721401 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a566f0ac-617b-4f46-a46d-faaae0391a93-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:12:15.723240 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:15.723208 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a566f0ac-617b-4f46-a46d-faaae0391a93-kube-api-access-k4pmq" pod "a566f0ac-617b-4f46-a46d-faaae0391a93" (UID: "a566f0ac-617b-4f46-a46d-faaae0391a93"). InnerVolumeSpecName "kube-api-access-k4pmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:15.723409 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:15.723379 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a566f0ac-617b-4f46-a46d-faaae0391a93-maas-api-xks-ca" pod "a566f0ac-617b-4f46-a46d-faaae0391a93" (UID: "a566f0ac-617b-4f46-a46d-faaae0391a93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:12:15.822960 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:15.822921 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4pmq\" (UniqueName: \"kubernetes.io/projected/a566f0ac-617b-4f46-a46d-faaae0391a93-kube-api-access-k4pmq\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:12:15.822960 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:15.822957 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a566f0ac-617b-4f46-a46d-faaae0391a93-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:12:16.476282 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:16.476238 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75abce-66k7k" Oct 30 13:12:16.476476 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:16.476246 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75abce-66k7k" event={"ID":"a566f0ac-617b-4f46-a46d-faaae0391a93","Type":"ContainerDied","Data":"a8f9c445c1141bcce57c7bc071b54d42d97e35b388f0531839113821707d33f1"} Oct 30 13:12:16.476476 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:16.476419 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8f9c445c1141bcce57c7bc071b54d42d97e35b388f0531839113821707d33f1" Oct 30 13:12:25.192318 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:12:25.192277 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:12:37.354345 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:12:37.354306 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:13:02.178305 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:02.178257 4931 scope.go:122] "RemoveContainer" containerID="f9ef3e48b4ebebd3ceeadf20cc81b07e460763bc761c480752c048aaf332e28a" Oct 30 13:13:02.188863 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:02.188834 4931 scope.go:122] "RemoveContainer" containerID="62e88201f1dabe6d3d89e41f39edecc2bcc3cf8277045bd8be286120387762ed" Oct 30 13:13:02.198363 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:02.198341 4931 scope.go:122] "RemoveContainer" containerID="8c7856d48f7d37f5f7a2d314389687689585844b4a1bb7cf9d4ee87214b9af16" Oct 30 13:13:07.361587 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:13:07.361543 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:13:07.803383 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:07.803350 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9074-71f11a"] Oct 30 13:13:07.804010 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:07.803994 4931 cpu_manager.go:414] "containerMap: removing container" podUID="a566f0ac-617b-4f46-a46d-faaae0391a93" containerName="revoke-keys" Oct 30 13:13:07.804061 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:07.804012 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a566f0ac-617b-4f46-a46d-faaae0391a93" containerName="revoke-keys" Oct 30 13:13:07.804133 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:07.804123 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="a566f0ac-617b-4f46-a46d-faaae0391a93" containerName="revoke-keys" Oct 30 13:13:07.809397 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:07.809369 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9074-71f11a" Oct 30 13:13:07.825542 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:07.825512 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8xhdg\"" Oct 30 13:13:07.826240 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:07.826211 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9074-71f11a"] Oct 30 13:13:07.990138 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:07.990095 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrtqh\" (UniqueName: \"kubernetes.io/projected/363a1522-03a1-4ac4-8b0d-b1505a906378-kube-api-access-rrtqh\") pod \"test-curl-9074-71f11a\" (UID: \"363a1522-03a1-4ac4-8b0d-b1505a906378\") " pod="opendatahub/test-curl-9074-71f11a" Oct 30 13:13:08.091861 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:08.091762 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrtqh\" (UniqueName: \"kubernetes.io/projected/363a1522-03a1-4ac4-8b0d-b1505a906378-kube-api-access-rrtqh\") pod \"test-curl-9074-71f11a\" (UID: \"363a1522-03a1-4ac4-8b0d-b1505a906378\") " pod="opendatahub/test-curl-9074-71f11a" Oct 30 13:13:08.116544 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:08.116511 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrtqh\" (UniqueName: \"kubernetes.io/projected/363a1522-03a1-4ac4-8b0d-b1505a906378-kube-api-access-rrtqh\") pod \"test-curl-9074-71f11a\" (UID: \"363a1522-03a1-4ac4-8b0d-b1505a906378\") " pod="opendatahub/test-curl-9074-71f11a" Oct 30 13:13:08.123822 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:08.123786 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9074-71f11a" Oct 30 13:13:08.435447 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:08.435417 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9074-71f11a"] Oct 30 13:13:08.435792 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:13:08.435502 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod363a1522_03a1_4ac4_8b0d_b1505a906378.slice/crio-7c347cd1d27531b378ee05571c0b01735e29f6b5561e1ba5ae19798be8e849fa WatchSource:0}: Error finding container 7c347cd1d27531b378ee05571c0b01735e29f6b5561e1ba5ae19798be8e849fa: Status 404 returned error can't find the container with id 7c347cd1d27531b378ee05571c0b01735e29f6b5561e1ba5ae19798be8e849fa Oct 30 13:13:09.270030 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:09.269995 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9074-71f11a" event={"ID":"363a1522-03a1-4ac4-8b0d-b1505a906378","Type":"ContainerStarted","Data":"5a9cb6f2c6a721c6ee05dc2fd2f10c7e0363b6fbc8cc3caf066592b67b547579"} Oct 30 13:13:09.270211 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:09.270044 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9074-71f11a" event={"ID":"363a1522-03a1-4ac4-8b0d-b1505a906378","Type":"ContainerStarted","Data":"7c347cd1d27531b378ee05571c0b01735e29f6b5561e1ba5ae19798be8e849fa"} Oct 30 13:13:09.338793 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:09.338730 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9074-71f11a" podStartSLOduration=2.338716555 podStartE2EDuration="2.338716555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:09.334782919 +0000 UTC m=+1687.938281393" watchObservedRunningTime="2026-10-30 13:13:09.338716555 +0000 UTC m=+1687.942215000" Oct 30 13:13:10.410086 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:10.410047 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9074-71f11a"] Oct 30 13:13:10.419454 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:10.419417 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9074-71f11a"] Oct 30 13:13:10.707075 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:10.707023 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-kk4l6"] Oct 30 13:13:10.708014 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:10.707988 4931 cpu_manager.go:414] "containerMap: removing container" podUID="363a1522-03a1-4ac4-8b0d-b1505a906378" containerName="test-curl-9074-71f11a" Oct 30 13:13:10.708089 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:10.708017 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="363a1522-03a1-4ac4-8b0d-b1505a906378" containerName="test-curl-9074-71f11a" Oct 30 13:13:10.708294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:10.708251 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="363a1522-03a1-4ac4-8b0d-b1505a906378" containerName="test-curl-9074-71f11a" Oct 30 13:13:10.712741 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:10.712715 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-kk4l6" Oct 30 13:13:10.735321 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:10.735294 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-kk4l6"] Oct 30 13:13:10.742581 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:10.742552 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:13:10.742699 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:10.742645 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:13:10.750822 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:10.750791 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e65a0bf-fdb0-452d-b9bd-a75372fcfba6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1c0e6c42-kk4l6\" (UID: \"2e65a0bf-fdb0-452d-b9bd-a75372fcfba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-kk4l6" Oct 30 13:13:10.750950 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:10.750931 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e65a0bf-fdb0-452d-b9bd-a75372fcfba6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1c0e6c42-kk4l6\" (UID: \"2e65a0bf-fdb0-452d-b9bd-a75372fcfba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-kk4l6" Oct 30 13:13:10.751095 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:10.751069 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s5hx\" (UniqueName: \"kubernetes.io/projected/2e65a0bf-fdb0-452d-b9bd-a75372fcfba6-kube-api-access-5s5hx\") pod \"maas-api-revoke-keys-e2e-mt-1c0e6c42-kk4l6\" (UID: \"2e65a0bf-fdb0-452d-b9bd-a75372fcfba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-kk4l6" Oct 30 13:13:10.852356 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:10.852318 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e65a0bf-fdb0-452d-b9bd-a75372fcfba6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1c0e6c42-kk4l6\" (UID: \"2e65a0bf-fdb0-452d-b9bd-a75372fcfba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-kk4l6" Oct 30 13:13:10.852543 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:10.852443 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s5hx\" (UniqueName: \"kubernetes.io/projected/2e65a0bf-fdb0-452d-b9bd-a75372fcfba6-kube-api-access-5s5hx\") pod \"maas-api-revoke-keys-e2e-mt-1c0e6c42-kk4l6\" (UID: \"2e65a0bf-fdb0-452d-b9bd-a75372fcfba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-kk4l6" Oct 30 13:13:10.852741 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:10.852722 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e65a0bf-fdb0-452d-b9bd-a75372fcfba6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1c0e6c42-kk4l6\" (UID: \"2e65a0bf-fdb0-452d-b9bd-a75372fcfba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-kk4l6" Oct 30 13:13:10.853019 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:10.853001 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e65a0bf-fdb0-452d-b9bd-a75372fcfba6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1c0e6c42-kk4l6\" (UID: \"2e65a0bf-fdb0-452d-b9bd-a75372fcfba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-kk4l6" Oct 30 13:13:10.855021 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:10.855001 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e65a0bf-fdb0-452d-b9bd-a75372fcfba6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1c0e6c42-kk4l6\" (UID: \"2e65a0bf-fdb0-452d-b9bd-a75372fcfba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-kk4l6" Oct 30 13:13:10.881777 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:10.881747 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s5hx\" (UniqueName: \"kubernetes.io/projected/2e65a0bf-fdb0-452d-b9bd-a75372fcfba6-kube-api-access-5s5hx\") pod \"maas-api-revoke-keys-e2e-mt-1c0e6c42-kk4l6\" (UID: \"2e65a0bf-fdb0-452d-b9bd-a75372fcfba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-kk4l6" Oct 30 13:13:11.027725 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:11.027645 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-kk4l6" Oct 30 13:13:11.285729 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:11.285615 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9074-71f11a" podUID="363a1522-03a1-4ac4-8b0d-b1505a906378" containerName="test-curl-9074-71f11a" containerID="cri-o://5a9cb6f2c6a721c6ee05dc2fd2f10c7e0363b6fbc8cc3caf066592b67b547579" gracePeriod=2 Oct 30 13:13:11.301807 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:13:11.301758 4931 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9074-71f11a\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" podUID="363a1522-03a1-4ac4-8b0d-b1505a906378" pod="opendatahub/test-curl-9074-71f11a" Oct 30 13:13:11.346412 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:13:11.346373 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e65a0bf_fdb0_452d_b9bd_a75372fcfba6.slice/crio-d01d91693d92cfd08e3733b45f3232c70e0f11ed5dbca392b487802c239aa7eb WatchSource:0}: Error finding container d01d91693d92cfd08e3733b45f3232c70e0f11ed5dbca392b487802c239aa7eb: Status 404 returned error can't find the container with id d01d91693d92cfd08e3733b45f3232c70e0f11ed5dbca392b487802c239aa7eb Oct 30 13:13:11.347004 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:11.346908 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-kk4l6"] Oct 30 13:13:11.525719 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:13:11.525680 4931 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9074-71f11a\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" podUID="363a1522-03a1-4ac4-8b0d-b1505a906378" pod="opendatahub/test-curl-9074-71f11a" Oct 30 13:13:11.601764 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:11.601725 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf"] Oct 30 13:13:11.608370 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:11.608344 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:13:11.643616 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:11.643583 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-11cfa6-openshift-default-dockercfg-dsqbb\"" Oct 30 13:13:11.651829 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:11.651797 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf"] Oct 30 13:13:11.665917 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:11.665873 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-credential-socket\") pod \"e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:13:11.666098 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:11.665994 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-workload-certs\") pod \"e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:13:11.666098 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:11.666088 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-workload-socket\") pod \"e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:13:11.666216 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:11.666191 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-istio-data\") pod \"e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:13:11.666400 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:11.666374 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-istio-podinfo\") pod \"e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:13:11.666496 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:11.666456 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-istio-envoy\") pod \"e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:13:11.666556 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:11.666503 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-istiod-ca-cert\") pod \"e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:13:11.666777 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:11.666754 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-istio-token\") pod \"e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:13:11.667157 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:11.666924 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm7sw\" (UniqueName: \"kubernetes.io/projected/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-kube-api-access-sm7sw\") pod \"e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:13:11.772290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:11.768171 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-istio-podinfo\") pod \"e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:13:11.772290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:11.768289 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-istio-envoy\") pod \"e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:13:11.772290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:11.768339 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-istiod-ca-cert\") pod \"e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:13:11.772290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:11.768387 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-istio-token\") pod \"e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:13:11.772290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:11.768509 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm7sw\" (UniqueName: \"kubernetes.io/projected/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-kube-api-access-sm7sw\") pod \"e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:13:11.772290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:11.768549 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-credential-socket\") pod \"e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:13:11.772290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:11.768647 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-workload-certs\") pod \"e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:13:11.772290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:11.768741 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-workload-socket\") pod \"e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:13:11.772290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:11.768782 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-istio-data\") pod \"e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:13:11.772959 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:11.772856 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-workload-certs\") pod \"e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:13:11.773283 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:11.773219 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-istiod-ca-cert\") pod \"e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:13:11.773439 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:11.773413 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-workload-socket\") pod \"e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:13:11.773516 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:11.773422 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-istio-data\") pod \"e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:13:11.773647 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:11.773613 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-credential-socket\") pod \"e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:13:11.773982 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:11.773957 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-istio-envoy\") pod \"e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:13:11.802322 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:11.802202 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-istio-podinfo\") pod \"e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:13:11.823358 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:11.819938 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-istio-token\") pod \"e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:13:11.826898 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:11.826829 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm7sw\" (UniqueName: \"kubernetes.io/projected/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-kube-api-access-sm7sw\") pod \"e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:13:11.928565 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:11.925331 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:13:12.295511 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:12.295471 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-kk4l6" event={"ID":"2e65a0bf-fdb0-452d-b9bd-a75372fcfba6","Type":"ContainerStarted","Data":"3be1ed0eccb9cc0943b0eba3ceae7d4dbace4d8976a3e731a5edf51e10c39218"} Oct 30 13:13:12.295734 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:12.295527 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-kk4l6" event={"ID":"2e65a0bf-fdb0-452d-b9bd-a75372fcfba6","Type":"ContainerStarted","Data":"d01d91693d92cfd08e3733b45f3232c70e0f11ed5dbca392b487802c239aa7eb"} Oct 30 13:13:12.342826 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:12.342796 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf"] Oct 30 13:13:12.343899 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:13:12.343852 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef8a128_1b22_4ef2_81ed_0a1b494393c3.slice/crio-2ce037ab67784ab8b32f52041e5a4ba783eb2dd80af610364221325ac529302f WatchSource:0}: Error finding container 2ce037ab67784ab8b32f52041e5a4ba783eb2dd80af610364221325ac529302f: Status 404 returned error can't find the container with id 2ce037ab67784ab8b32f52041e5a4ba783eb2dd80af610364221325ac529302f Oct 30 13:13:12.347732 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:12.347692 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:13:12.347851 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:12.347783 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:13:12.347851 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:12.347829 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:13:12.355243 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:12.355201 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-kk4l6" podStartSLOduration=2.355188225 podStartE2EDuration="2.355188225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:12.350034077 +0000 UTC m=+1690.953532523" watchObservedRunningTime="2026-10-30 13:13:12.355188225 +0000 UTC m=+1690.958686671" Oct 30 13:13:13.305434 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:13.305395 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" event={"ID":"9ef8a128-1b22-4ef2-81ed-0a1b494393c3","Type":"ContainerStarted","Data":"4e1eeee4d82f56f22f06c4a0c2ed45a49a1fb852be0f2d410ed0f0ed98784c6e"} Oct 30 13:13:13.305856 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:13.305500 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" event={"ID":"9ef8a128-1b22-4ef2-81ed-0a1b494393c3","Type":"ContainerStarted","Data":"2ce037ab67784ab8b32f52041e5a4ba783eb2dd80af610364221325ac529302f"} Oct 30 13:13:13.307361 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:13.307336 4931 generic.go:356] "Generic (PLEG): container finished" podID="2e65a0bf-fdb0-452d-b9bd-a75372fcfba6" containerID="3be1ed0eccb9cc0943b0eba3ceae7d4dbace4d8976a3e731a5edf51e10c39218" exitCode=0 Oct 30 13:13:13.307509 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:13.307427 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-kk4l6" event={"ID":"2e65a0bf-fdb0-452d-b9bd-a75372fcfba6","Type":"ContainerDied","Data":"3be1ed0eccb9cc0943b0eba3ceae7d4dbace4d8976a3e731a5edf51e10c39218"} Oct 30 13:13:13.383410 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:13.383352 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" podStartSLOduration=2.383334407 podStartE2EDuration="2.383334407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:13.378354205 +0000 UTC m=+1691.981852651" watchObservedRunningTime="2026-10-30 13:13:13.383334407 +0000 UTC m=+1691.986832854" Oct 30 13:13:13.560406 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:13.560310 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9074-71f11a" Oct 30 13:13:13.648718 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:13.648685 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/363a1522-03a1-4ac4-8b0d-b1505a906378-kube-api-access-rrtqh\" (UniqueName: \"kubernetes.io/projected/363a1522-03a1-4ac4-8b0d-b1505a906378-kube-api-access-rrtqh\") pod \"363a1522-03a1-4ac4-8b0d-b1505a906378\" (UID: \"363a1522-03a1-4ac4-8b0d-b1505a906378\") " Oct 30 13:13:13.651092 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:13.651057 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363a1522-03a1-4ac4-8b0d-b1505a906378-kube-api-access-rrtqh" pod "363a1522-03a1-4ac4-8b0d-b1505a906378" (UID: "363a1522-03a1-4ac4-8b0d-b1505a906378"). InnerVolumeSpecName "kube-api-access-rrtqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:13.751532 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:13.751499 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrtqh\" (UniqueName: \"kubernetes.io/projected/363a1522-03a1-4ac4-8b0d-b1505a906378-kube-api-access-rrtqh\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:13:13.926284 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:13.926188 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:13:13.932110 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:13.932083 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:13:14.315737 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:14.315698 4931 generic.go:356] "Generic (PLEG): container finished" podID="363a1522-03a1-4ac4-8b0d-b1505a906378" containerID="5a9cb6f2c6a721c6ee05dc2fd2f10c7e0363b6fbc8cc3caf066592b67b547579" exitCode=137 Oct 30 13:13:14.316211 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:14.315744 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9074-71f11a" Oct 30 13:13:14.316211 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:14.315797 4931 scope.go:122] "RemoveContainer" containerID="5a9cb6f2c6a721c6ee05dc2fd2f10c7e0363b6fbc8cc3caf066592b67b547579" Oct 30 13:13:14.316473 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:14.316298 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:13:14.317708 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:14.317680 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:13:14.326459 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:14.326422 4931 scope.go:122] "RemoveContainer" containerID="5a9cb6f2c6a721c6ee05dc2fd2f10c7e0363b6fbc8cc3caf066592b67b547579" Oct 30 13:13:14.326752 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:13:14.326729 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a9cb6f2c6a721c6ee05dc2fd2f10c7e0363b6fbc8cc3caf066592b67b547579\": container with ID starting with 5a9cb6f2c6a721c6ee05dc2fd2f10c7e0363b6fbc8cc3caf066592b67b547579 not found: ID does not exist" containerID="5a9cb6f2c6a721c6ee05dc2fd2f10c7e0363b6fbc8cc3caf066592b67b547579" Oct 30 13:13:14.326835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:14.326765 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a9cb6f2c6a721c6ee05dc2fd2f10c7e0363b6fbc8cc3caf066592b67b547579"} err="failed to get container status \"5a9cb6f2c6a721c6ee05dc2fd2f10c7e0363b6fbc8cc3caf066592b67b547579\": rpc error: code = NotFound desc = could not find container \"5a9cb6f2c6a721c6ee05dc2fd2f10c7e0363b6fbc8cc3caf066592b67b547579\": container with ID starting with 5a9cb6f2c6a721c6ee05dc2fd2f10c7e0363b6fbc8cc3caf066592b67b547579 not found: ID does not exist" Oct 30 13:13:14.393968 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:13:14.393928 4931 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9074-71f11a\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" podUID="363a1522-03a1-4ac4-8b0d-b1505a906378" pod="opendatahub/test-curl-9074-71f11a" Oct 30 13:13:14.513772 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:14.513742 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-kk4l6" Oct 30 13:13:14.528739 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:13:14.528706 4931 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9074-71f11a\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" podUID="363a1522-03a1-4ac4-8b0d-b1505a906378" pod="opendatahub/test-curl-9074-71f11a" Oct 30 13:13:14.665504 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:14.665429 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e65a0bf-fdb0-452d-b9bd-a75372fcfba6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e65a0bf-fdb0-452d-b9bd-a75372fcfba6-maas-api-openshift-ca\") pod \"2e65a0bf-fdb0-452d-b9bd-a75372fcfba6\" (UID: \"2e65a0bf-fdb0-452d-b9bd-a75372fcfba6\") " Oct 30 13:13:14.665741 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:14.665718 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e65a0bf-fdb0-452d-b9bd-a75372fcfba6-kube-api-access-5s5hx\" (UniqueName: \"kubernetes.io/projected/2e65a0bf-fdb0-452d-b9bd-a75372fcfba6-kube-api-access-5s5hx\") pod \"2e65a0bf-fdb0-452d-b9bd-a75372fcfba6\" (UID: \"2e65a0bf-fdb0-452d-b9bd-a75372fcfba6\") " Oct 30 13:13:14.665824 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:14.665808 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e65a0bf-fdb0-452d-b9bd-a75372fcfba6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e65a0bf-fdb0-452d-b9bd-a75372fcfba6-maas-api-xks-ca\") pod \"2e65a0bf-fdb0-452d-b9bd-a75372fcfba6\" (UID: \"2e65a0bf-fdb0-452d-b9bd-a75372fcfba6\") " Oct 30 13:13:14.665943 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:14.665813 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e65a0bf-fdb0-452d-b9bd-a75372fcfba6-maas-api-openshift-ca" pod "2e65a0bf-fdb0-452d-b9bd-a75372fcfba6" (UID: "2e65a0bf-fdb0-452d-b9bd-a75372fcfba6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:14.666928 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:14.666900 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e65a0bf-fdb0-452d-b9bd-a75372fcfba6-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:13:14.668040 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:14.668019 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e65a0bf-fdb0-452d-b9bd-a75372fcfba6-kube-api-access-5s5hx" pod "2e65a0bf-fdb0-452d-b9bd-a75372fcfba6" (UID: "2e65a0bf-fdb0-452d-b9bd-a75372fcfba6"). InnerVolumeSpecName "kube-api-access-5s5hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:14.668135 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:14.668025 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e65a0bf-fdb0-452d-b9bd-a75372fcfba6-maas-api-xks-ca" pod "2e65a0bf-fdb0-452d-b9bd-a75372fcfba6" (UID: "2e65a0bf-fdb0-452d-b9bd-a75372fcfba6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:13:14.768531 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:14.768501 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s5hx\" (UniqueName: \"kubernetes.io/projected/2e65a0bf-fdb0-452d-b9bd-a75372fcfba6-kube-api-access-5s5hx\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:13:14.768531 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:14.768528 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e65a0bf-fdb0-452d-b9bd-a75372fcfba6-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:13:15.323994 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:15.323955 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-kk4l6" Oct 30 13:13:15.324481 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:15.323988 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-kk4l6" event={"ID":"2e65a0bf-fdb0-452d-b9bd-a75372fcfba6","Type":"ContainerDied","Data":"d01d91693d92cfd08e3733b45f3232c70e0f11ed5dbca392b487802c239aa7eb"} Oct 30 13:13:15.324481 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:15.324049 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d01d91693d92cfd08e3733b45f3232c70e0f11ed5dbca392b487802c239aa7eb" Oct 30 13:13:15.349126 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:13:15.349094 4931 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9074-71f11a\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" podUID="363a1522-03a1-4ac4-8b0d-b1505a906378" pod="opendatahub/test-curl-9074-71f11a" Oct 30 13:13:15.514978 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:15.514946 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="363a1522-03a1-4ac4-8b0d-b1505a906378" path="/var/lib/kubelet/pods/363a1522-03a1-4ac4-8b0d-b1505a906378/volumes" Oct 30 13:13:27.867444 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:27.867407 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3160a-ww8z2"] Oct 30 13:13:27.868064 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:27.868049 4931 cpu_manager.go:414] "containerMap: removing container" podUID="2e65a0bf-fdb0-452d-b9bd-a75372fcfba6" containerName="revoke-keys" Oct 30 13:13:27.868109 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:27.868065 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e65a0bf-fdb0-452d-b9bd-a75372fcfba6" containerName="revoke-keys" Oct 30 13:13:27.868186 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:27.868176 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e65a0bf-fdb0-452d-b9bd-a75372fcfba6" containerName="revoke-keys" Oct 30 13:13:27.872007 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:27.871986 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3160a-ww8z2" Oct 30 13:13:27.891959 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:27.891930 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3160a-ww8z2"] Oct 30 13:13:27.896045 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:27.896020 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:13:27.896340 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:27.896320 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:13:27.998133 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:27.998093 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3242035b-ebb3-45dc-9af0-c0ce730be9ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f3160a-ww8z2\" (UID: \"3242035b-ebb3-45dc-9af0-c0ce730be9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3160a-ww8z2" Oct 30 13:13:27.998366 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:27.998339 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6q85\" (UniqueName: \"kubernetes.io/projected/3242035b-ebb3-45dc-9af0-c0ce730be9ca-kube-api-access-m6q85\") pod \"maas-api-revoke-keys-e2e-api-a-f3160a-ww8z2\" (UID: \"3242035b-ebb3-45dc-9af0-c0ce730be9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3160a-ww8z2" Oct 30 13:13:27.998419 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:27.998397 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3242035b-ebb3-45dc-9af0-c0ce730be9ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f3160a-ww8z2\" (UID: \"3242035b-ebb3-45dc-9af0-c0ce730be9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3160a-ww8z2" Oct 30 13:13:28.099994 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:28.099955 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3242035b-ebb3-45dc-9af0-c0ce730be9ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f3160a-ww8z2\" (UID: \"3242035b-ebb3-45dc-9af0-c0ce730be9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3160a-ww8z2" Oct 30 13:13:28.100174 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:28.100068 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6q85\" (UniqueName: \"kubernetes.io/projected/3242035b-ebb3-45dc-9af0-c0ce730be9ca-kube-api-access-m6q85\") pod \"maas-api-revoke-keys-e2e-api-a-f3160a-ww8z2\" (UID: \"3242035b-ebb3-45dc-9af0-c0ce730be9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3160a-ww8z2" Oct 30 13:13:28.100174 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:28.100098 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3242035b-ebb3-45dc-9af0-c0ce730be9ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f3160a-ww8z2\" (UID: \"3242035b-ebb3-45dc-9af0-c0ce730be9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3160a-ww8z2" Oct 30 13:13:28.100678 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:28.100656 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3242035b-ebb3-45dc-9af0-c0ce730be9ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f3160a-ww8z2\" (UID: \"3242035b-ebb3-45dc-9af0-c0ce730be9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3160a-ww8z2" Oct 30 13:13:28.102620 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:28.102595 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3242035b-ebb3-45dc-9af0-c0ce730be9ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f3160a-ww8z2\" (UID: \"3242035b-ebb3-45dc-9af0-c0ce730be9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3160a-ww8z2" Oct 30 13:13:28.124311 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:28.124245 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6q85\" (UniqueName: \"kubernetes.io/projected/3242035b-ebb3-45dc-9af0-c0ce730be9ca-kube-api-access-m6q85\") pod \"maas-api-revoke-keys-e2e-api-a-f3160a-ww8z2\" (UID: \"3242035b-ebb3-45dc-9af0-c0ce730be9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3160a-ww8z2" Oct 30 13:13:28.187291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:28.186573 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3160a-ww8z2" Oct 30 13:13:28.510428 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:13:28.510396 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3242035b_ebb3_45dc_9af0_c0ce730be9ca.slice/crio-8ec71efbc584d283c7ff4c85b2fbbce54e07f4d4c9901d804ecb1d15119e301e WatchSource:0}: Error finding container 8ec71efbc584d283c7ff4c85b2fbbce54e07f4d4c9901d804ecb1d15119e301e: Status 404 returned error can't find the container with id 8ec71efbc584d283c7ff4c85b2fbbce54e07f4d4c9901d804ecb1d15119e301e Oct 30 13:13:28.510731 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:28.510704 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3160a-ww8z2"] Oct 30 13:13:28.577716 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:28.577684 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3160a-ww8z2" event={"ID":"3242035b-ebb3-45dc-9af0-c0ce730be9ca","Type":"ContainerStarted","Data":"8ec71efbc584d283c7ff4c85b2fbbce54e07f4d4c9901d804ecb1d15119e301e"} Oct 30 13:13:29.587086 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:29.587049 4931 generic.go:356] "Generic (PLEG): container finished" podID="3242035b-ebb3-45dc-9af0-c0ce730be9ca" containerID="f38acf38fe3a892125cc703a1ac57fb4e74fcaf42cbd4b9cf6ef27f55ea5dfca" exitCode=0 Oct 30 13:13:29.587562 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:29.587128 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3160a-ww8z2" event={"ID":"3242035b-ebb3-45dc-9af0-c0ce730be9ca","Type":"ContainerDied","Data":"f38acf38fe3a892125cc703a1ac57fb4e74fcaf42cbd4b9cf6ef27f55ea5dfca"} Oct 30 13:13:30.170132 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:30.170096 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:13:30.773853 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:30.773827 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3160a-ww8z2" Oct 30 13:13:30.948368 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:30.948337 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3242035b-ebb3-45dc-9af0-c0ce730be9ca-kube-api-access-m6q85\" (UniqueName: \"kubernetes.io/projected/3242035b-ebb3-45dc-9af0-c0ce730be9ca-kube-api-access-m6q85\") pod \"3242035b-ebb3-45dc-9af0-c0ce730be9ca\" (UID: \"3242035b-ebb3-45dc-9af0-c0ce730be9ca\") " Oct 30 13:13:30.948574 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:30.948559 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3242035b-ebb3-45dc-9af0-c0ce730be9ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3242035b-ebb3-45dc-9af0-c0ce730be9ca-maas-api-openshift-ca\") pod \"3242035b-ebb3-45dc-9af0-c0ce730be9ca\" (UID: \"3242035b-ebb3-45dc-9af0-c0ce730be9ca\") " Oct 30 13:13:30.948665 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:30.948640 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3242035b-ebb3-45dc-9af0-c0ce730be9ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3242035b-ebb3-45dc-9af0-c0ce730be9ca-maas-api-xks-ca\") pod \"3242035b-ebb3-45dc-9af0-c0ce730be9ca\" (UID: \"3242035b-ebb3-45dc-9af0-c0ce730be9ca\") " Oct 30 13:13:30.948987 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:30.948950 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3242035b-ebb3-45dc-9af0-c0ce730be9ca-maas-api-openshift-ca" pod "3242035b-ebb3-45dc-9af0-c0ce730be9ca" (UID: "3242035b-ebb3-45dc-9af0-c0ce730be9ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:30.949304 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:30.949252 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3242035b-ebb3-45dc-9af0-c0ce730be9ca-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:13:30.950632 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:30.950608 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3242035b-ebb3-45dc-9af0-c0ce730be9ca-kube-api-access-m6q85" pod "3242035b-ebb3-45dc-9af0-c0ce730be9ca" (UID: "3242035b-ebb3-45dc-9af0-c0ce730be9ca"). InnerVolumeSpecName "kube-api-access-m6q85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:30.950731 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:30.950673 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3242035b-ebb3-45dc-9af0-c0ce730be9ca-maas-api-xks-ca" pod "3242035b-ebb3-45dc-9af0-c0ce730be9ca" (UID: "3242035b-ebb3-45dc-9af0-c0ce730be9ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:13:31.050985 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:31.050949 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3242035b-ebb3-45dc-9af0-c0ce730be9ca-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:13:31.050985 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:31.050979 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6q85\" (UniqueName: \"kubernetes.io/projected/3242035b-ebb3-45dc-9af0-c0ce730be9ca-kube-api-access-m6q85\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:13:31.603437 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:31.603406 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3160a-ww8z2" Oct 30 13:13:31.603613 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:31.603459 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3160a-ww8z2" event={"ID":"3242035b-ebb3-45dc-9af0-c0ce730be9ca","Type":"ContainerDied","Data":"8ec71efbc584d283c7ff4c85b2fbbce54e07f4d4c9901d804ecb1d15119e301e"} Oct 30 13:13:31.603613 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:31.603508 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ec71efbc584d283c7ff4c85b2fbbce54e07f4d4c9901d804ecb1d15119e301e" Oct 30 13:13:37.353791 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:13:37.353754 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:13:41.776477 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:41.776438 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75abce-66k7k"] Oct 30 13:13:41.803465 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:41.803430 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75abce-66k7k"] Oct 30 13:13:43.514876 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:43.514839 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a566f0ac-617b-4f46-a46d-faaae0391a93" path="/var/lib/kubelet/pods/a566f0ac-617b-4f46-a46d-faaae0391a93/volumes" Oct 30 13:13:53.065332 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:53.065295 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-kk4l6"] Oct 30 13:13:53.078334 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:53.078298 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-kk4l6"] Oct 30 13:13:53.517696 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:53.517661 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e65a0bf-fdb0-452d-b9bd-a75372fcfba6" path="/var/lib/kubelet/pods/2e65a0bf-fdb0-452d-b9bd-a75372fcfba6/volumes" Oct 30 13:13:53.598821 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:53.598783 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31e10b-75k7t"] Oct 30 13:13:53.599842 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:53.599820 4931 cpu_manager.go:414] "containerMap: removing container" podUID="3242035b-ebb3-45dc-9af0-c0ce730be9ca" containerName="revoke-keys" Oct 30 13:13:53.599915 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:53.599848 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3242035b-ebb3-45dc-9af0-c0ce730be9ca" containerName="revoke-keys" Oct 30 13:13:53.600018 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:53.600005 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="3242035b-ebb3-45dc-9af0-c0ce730be9ca" containerName="revoke-keys" Oct 30 13:13:53.604570 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:53.604546 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31e10b-75k7t" Oct 30 13:13:53.621676 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:53.621651 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:13:53.641997 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:53.641966 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31e10b-75k7t"] Oct 30 13:13:53.691152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:53.691113 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3037fde1-6ac2-4d15-8ecc-354dee56569c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-31e10b-75k7t\" (UID: \"3037fde1-6ac2-4d15-8ecc-354dee56569c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31e10b-75k7t" Oct 30 13:13:53.691343 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:53.691177 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87zgc\" (UniqueName: \"kubernetes.io/projected/3037fde1-6ac2-4d15-8ecc-354dee56569c-kube-api-access-87zgc\") pod \"maas-api-revoke-keys-e2e-api-b-31e10b-75k7t\" (UID: \"3037fde1-6ac2-4d15-8ecc-354dee56569c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31e10b-75k7t" Oct 30 13:13:53.691453 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:53.691430 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3037fde1-6ac2-4d15-8ecc-354dee56569c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-31e10b-75k7t\" (UID: \"3037fde1-6ac2-4d15-8ecc-354dee56569c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31e10b-75k7t" Oct 30 13:13:53.721669 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:53.721639 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:13:53.815435 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:53.798678 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3037fde1-6ac2-4d15-8ecc-354dee56569c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-31e10b-75k7t\" (UID: \"3037fde1-6ac2-4d15-8ecc-354dee56569c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31e10b-75k7t" Oct 30 13:13:53.838287 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:53.815978 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3037fde1-6ac2-4d15-8ecc-354dee56569c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-31e10b-75k7t\" (UID: \"3037fde1-6ac2-4d15-8ecc-354dee56569c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31e10b-75k7t" Oct 30 13:13:53.857305 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:53.857250 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87zgc\" (UniqueName: \"kubernetes.io/projected/3037fde1-6ac2-4d15-8ecc-354dee56569c-kube-api-access-87zgc\") pod \"maas-api-revoke-keys-e2e-api-b-31e10b-75k7t\" (UID: \"3037fde1-6ac2-4d15-8ecc-354dee56569c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31e10b-75k7t" Oct 30 13:13:53.857864 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:53.802621 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3037fde1-6ac2-4d15-8ecc-354dee56569c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-31e10b-75k7t\" (UID: \"3037fde1-6ac2-4d15-8ecc-354dee56569c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31e10b-75k7t" Oct 30 13:13:53.864876 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:53.864847 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3037fde1-6ac2-4d15-8ecc-354dee56569c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-31e10b-75k7t\" (UID: \"3037fde1-6ac2-4d15-8ecc-354dee56569c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31e10b-75k7t" Oct 30 13:13:53.916301 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:53.913419 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87zgc\" (UniqueName: \"kubernetes.io/projected/3037fde1-6ac2-4d15-8ecc-354dee56569c-kube-api-access-87zgc\") pod \"maas-api-revoke-keys-e2e-api-b-31e10b-75k7t\" (UID: \"3037fde1-6ac2-4d15-8ecc-354dee56569c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31e10b-75k7t" Oct 30 13:13:53.921790 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:53.920542 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31e10b-75k7t" Oct 30 13:13:54.270368 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:54.270322 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31e10b-75k7t"] Oct 30 13:13:54.275314 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:13:54.275227 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3037fde1_6ac2_4d15_8ecc_354dee56569c.slice/crio-a54219d097275178f887bb0de21dad251d4556c3fefc2694a455cf4b9ae38a38 WatchSource:0}: Error finding container a54219d097275178f887bb0de21dad251d4556c3fefc2694a455cf4b9ae38a38: Status 404 returned error can't find the container with id a54219d097275178f887bb0de21dad251d4556c3fefc2694a455cf4b9ae38a38 Oct 30 13:13:54.953639 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:54.953608 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31e10b-75k7t" event={"ID":"3037fde1-6ac2-4d15-8ecc-354dee56569c","Type":"ContainerStarted","Data":"d2937ccb41823d953519e3a1272f0ba27ba2cdda6fb163e7131f1e7e3e80a4df"} Oct 30 13:13:54.953826 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:54.953660 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31e10b-75k7t" event={"ID":"3037fde1-6ac2-4d15-8ecc-354dee56569c","Type":"ContainerStarted","Data":"a54219d097275178f887bb0de21dad251d4556c3fefc2694a455cf4b9ae38a38"} Oct 30 13:13:55.004125 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:55.004064 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31e10b-75k7t" podStartSLOduration=2.00404709 podStartE2EDuration="2.00404709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:55.000674482 +0000 UTC m=+1733.604172961" watchObservedRunningTime="2026-10-30 13:13:55.00404709 +0000 UTC m=+1733.607545535" Oct 30 13:13:55.963028 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:55.962993 4931 generic.go:356] "Generic (PLEG): container finished" podID="3037fde1-6ac2-4d15-8ecc-354dee56569c" containerID="d2937ccb41823d953519e3a1272f0ba27ba2cdda6fb163e7131f1e7e3e80a4df" exitCode=0 Oct 30 13:13:55.963429 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:55.963070 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31e10b-75k7t" event={"ID":"3037fde1-6ac2-4d15-8ecc-354dee56569c","Type":"ContainerDied","Data":"d2937ccb41823d953519e3a1272f0ba27ba2cdda6fb163e7131f1e7e3e80a4df"} Oct 30 13:13:57.139028 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:57.139000 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31e10b-75k7t" Oct 30 13:13:57.204897 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:57.204862 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3037fde1-6ac2-4d15-8ecc-354dee56569c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3037fde1-6ac2-4d15-8ecc-354dee56569c-maas-api-xks-ca\") pod \"3037fde1-6ac2-4d15-8ecc-354dee56569c\" (UID: \"3037fde1-6ac2-4d15-8ecc-354dee56569c\") " Oct 30 13:13:57.205081 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:57.205006 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3037fde1-6ac2-4d15-8ecc-354dee56569c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3037fde1-6ac2-4d15-8ecc-354dee56569c-maas-api-openshift-ca\") pod \"3037fde1-6ac2-4d15-8ecc-354dee56569c\" (UID: \"3037fde1-6ac2-4d15-8ecc-354dee56569c\") " Oct 30 13:13:57.205180 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:57.205162 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3037fde1-6ac2-4d15-8ecc-354dee56569c-kube-api-access-87zgc\" (UniqueName: \"kubernetes.io/projected/3037fde1-6ac2-4d15-8ecc-354dee56569c-kube-api-access-87zgc\") pod \"3037fde1-6ac2-4d15-8ecc-354dee56569c\" (UID: \"3037fde1-6ac2-4d15-8ecc-354dee56569c\") " Oct 30 13:13:57.205515 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:57.205483 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3037fde1-6ac2-4d15-8ecc-354dee56569c-maas-api-openshift-ca" pod "3037fde1-6ac2-4d15-8ecc-354dee56569c" (UID: "3037fde1-6ac2-4d15-8ecc-354dee56569c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:57.205976 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:57.205950 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3037fde1-6ac2-4d15-8ecc-354dee56569c-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:13:57.207197 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:57.207163 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3037fde1-6ac2-4d15-8ecc-354dee56569c-maas-api-xks-ca" pod "3037fde1-6ac2-4d15-8ecc-354dee56569c" (UID: "3037fde1-6ac2-4d15-8ecc-354dee56569c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:13:57.208494 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:57.208465 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3037fde1-6ac2-4d15-8ecc-354dee56569c-kube-api-access-87zgc" pod "3037fde1-6ac2-4d15-8ecc-354dee56569c" (UID: "3037fde1-6ac2-4d15-8ecc-354dee56569c"). InnerVolumeSpecName "kube-api-access-87zgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:57.307246 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:57.307207 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3037fde1-6ac2-4d15-8ecc-354dee56569c-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:13:57.307246 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:57.307238 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87zgc\" (UniqueName: \"kubernetes.io/projected/3037fde1-6ac2-4d15-8ecc-354dee56569c-kube-api-access-87zgc\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:13:57.980364 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:57.980336 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31e10b-75k7t" Oct 30 13:13:57.980534 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:57.980336 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31e10b-75k7t" event={"ID":"3037fde1-6ac2-4d15-8ecc-354dee56569c","Type":"ContainerDied","Data":"a54219d097275178f887bb0de21dad251d4556c3fefc2694a455cf4b9ae38a38"} Oct 30 13:13:57.980534 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:57.980465 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a54219d097275178f887bb0de21dad251d4556c3fefc2694a455cf4b9ae38a38" Oct 30 13:13:58.579229 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:13:58.579196 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs"] Oct 30 13:14:02.236809 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:02.236772 4931 scope.go:122] "RemoveContainer" containerID="9e6003ba488b867845bc7dc42d7e6ca45940185432dc3c2654ab77d894d42625" Oct 30 13:14:07.353629 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:14:07.353594 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:14:13.603280 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:13.603166 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" podUID="fb4d115a-ce18-42fe-a26e-d28f0cbadd1e" containerName="main" containerID="cri-o://4aaa5fc829c5b89ad97986542dbf5df1bbb4deee1a8a76270c5fe1d6c47c49ed" gracePeriod=45 Oct 30 13:14:13.784166 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:13.784128 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:14:14.069213 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:14.069185 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" Oct 30 13:14:14.097980 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:14.097953 4931 generic.go:356] "Generic (PLEG): container finished" podID="fb4d115a-ce18-42fe-a26e-d28f0cbadd1e" containerID="4aaa5fc829c5b89ad97986542dbf5df1bbb4deee1a8a76270c5fe1d6c47c49ed" exitCode=0 Oct 30 13:14:14.098155 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:14.098001 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" Oct 30 13:14:14.098155 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:14.098035 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" event={"ID":"fb4d115a-ce18-42fe-a26e-d28f0cbadd1e","Type":"ContainerDied","Data":"4aaa5fc829c5b89ad97986542dbf5df1bbb4deee1a8a76270c5fe1d6c47c49ed"} Oct 30 13:14:14.098155 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:14.098077 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" event={"ID":"fb4d115a-ce18-42fe-a26e-d28f0cbadd1e","Type":"ContainerDied","Data":"a3612114dec0a23706fb7f8b93df450c468260f759c75b7d0a33eedf1e959bea"} Oct 30 13:14:14.098155 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:14.098096 4931 scope.go:122] "RemoveContainer" containerID="4aaa5fc829c5b89ad97986542dbf5df1bbb4deee1a8a76270c5fe1d6c47c49ed" Oct 30 13:14:14.108234 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:14.108217 4931 scope.go:122] "RemoveContainer" containerID="4aaa5fc829c5b89ad97986542dbf5df1bbb4deee1a8a76270c5fe1d6c47c49ed" Oct 30 13:14:14.108479 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:14:14.108459 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aaa5fc829c5b89ad97986542dbf5df1bbb4deee1a8a76270c5fe1d6c47c49ed\": container with ID starting with 4aaa5fc829c5b89ad97986542dbf5df1bbb4deee1a8a76270c5fe1d6c47c49ed not found: ID does not exist" containerID="4aaa5fc829c5b89ad97986542dbf5df1bbb4deee1a8a76270c5fe1d6c47c49ed" Oct 30 13:14:14.108581 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:14.108486 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aaa5fc829c5b89ad97986542dbf5df1bbb4deee1a8a76270c5fe1d6c47c49ed"} err="failed to get container status \"4aaa5fc829c5b89ad97986542dbf5df1bbb4deee1a8a76270c5fe1d6c47c49ed\": rpc error: code = NotFound desc = could not find container \"4aaa5fc829c5b89ad97986542dbf5df1bbb4deee1a8a76270c5fe1d6c47c49ed\": container with ID starting with 4aaa5fc829c5b89ad97986542dbf5df1bbb4deee1a8a76270c5fe1d6c47c49ed not found: ID does not exist" Oct 30 13:14:14.222158 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:14.222122 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-tmp-dir\") pod \"fb4d115a-ce18-42fe-a26e-d28f0cbadd1e\" (UID: \"fb4d115a-ce18-42fe-a26e-d28f0cbadd1e\") " Oct 30 13:14:14.222394 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:14.222206 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-dshm\") pod \"fb4d115a-ce18-42fe-a26e-d28f0cbadd1e\" (UID: \"fb4d115a-ce18-42fe-a26e-d28f0cbadd1e\") " Oct 30 13:14:14.222469 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:14.222448 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-kube-api-access-pnmnx\" (UniqueName: \"kubernetes.io/projected/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-kube-api-access-pnmnx\") pod \"fb4d115a-ce18-42fe-a26e-d28f0cbadd1e\" (UID: \"fb4d115a-ce18-42fe-a26e-d28f0cbadd1e\") " Oct 30 13:14:14.222526 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:14.222463 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-tmp-dir" pod "fb4d115a-ce18-42fe-a26e-d28f0cbadd1e" (UID: "fb4d115a-ce18-42fe-a26e-d28f0cbadd1e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:14:14.222526 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:14.222515 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-home\" (UniqueName: \"kubernetes.io/empty-dir/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-home\") pod \"fb4d115a-ce18-42fe-a26e-d28f0cbadd1e\" (UID: \"fb4d115a-ce18-42fe-a26e-d28f0cbadd1e\") " Oct 30 13:14:14.222643 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:14.222573 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-model-cache\") pod \"fb4d115a-ce18-42fe-a26e-d28f0cbadd1e\" (UID: \"fb4d115a-ce18-42fe-a26e-d28f0cbadd1e\") " Oct 30 13:14:14.222697 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:14.222686 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-tls-certs\") pod \"fb4d115a-ce18-42fe-a26e-d28f0cbadd1e\" (UID: \"fb4d115a-ce18-42fe-a26e-d28f0cbadd1e\") " Oct 30 13:14:14.222804 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:14.222781 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-home" pod "fb4d115a-ce18-42fe-a26e-d28f0cbadd1e" (UID: "fb4d115a-ce18-42fe-a26e-d28f0cbadd1e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:14:14.222993 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:14.222975 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-model-cache" pod "fb4d115a-ce18-42fe-a26e-d28f0cbadd1e" (UID: "fb4d115a-ce18-42fe-a26e-d28f0cbadd1e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:14:14.223394 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:14.223370 4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-tmp-dir\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:14:14.223470 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:14.223400 4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-home\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:14:14.223470 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:14.223414 4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-model-cache\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:14:14.225034 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:14.225005 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-tls-certs" pod "fb4d115a-ce18-42fe-a26e-d28f0cbadd1e" (UID: "fb4d115a-ce18-42fe-a26e-d28f0cbadd1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:14:14.225242 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:14.225224 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-kube-api-access-pnmnx" pod "fb4d115a-ce18-42fe-a26e-d28f0cbadd1e" (UID: "fb4d115a-ce18-42fe-a26e-d28f0cbadd1e"). InnerVolumeSpecName "kube-api-access-pnmnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:14:14.225384 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:14.225359 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-dshm" pod "fb4d115a-ce18-42fe-a26e-d28f0cbadd1e" (UID: "fb4d115a-ce18-42fe-a26e-d28f0cbadd1e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:14:14.324689 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:14.324611 4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-tls-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:14:14.324689 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:14.324640 4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-dshm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:14:14.324689 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:14.324650 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnmnx\" (UniqueName: \"kubernetes.io/projected/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e-kube-api-access-pnmnx\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:14:14.450652 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:14.450624 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs"] Oct 30 13:14:14.463492 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:14.463462 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs"] Oct 30 13:14:14.925451 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:14.925403 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-1c0e6c42/e2e-lifecycle-model-1c0e6c42-kserve-7b69b66945-7tgjs" podUID="fb4d115a-ce18-42fe-a26e-d28f0cbadd1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded" Oct 30 13:14:15.531463 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:15.527398 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb4d115a-ce18-42fe-a26e-d28f0cbadd1e" path="/var/lib/kubelet/pods/fb4d115a-ce18-42fe-a26e-d28f0cbadd1e/volumes" Oct 30 13:14:26.970062 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:26.970025 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw"] Oct 30 13:14:26.970617 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:26.970225 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" podUID="a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63" containerName="istio-proxy" containerID="cri-o://550ee4479e76ad0efb2952086dd83b1277d42eb66ab1adc0c5d524053170dfb1" gracePeriod=30 Oct 30 13:14:32.245390 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:32.245357 4931 generic.go:356] "Generic (PLEG): container finished" podID="a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63" containerID="550ee4479e76ad0efb2952086dd83b1277d42eb66ab1adc0c5d524053170dfb1" exitCode=0 Oct 30 13:14:32.245705 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:32.245422 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" event={"ID":"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63","Type":"ContainerDied","Data":"550ee4479e76ad0efb2952086dd83b1277d42eb66ab1adc0c5d524053170dfb1"} Oct 30 13:14:32.245705 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:32.245473 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" event={"ID":"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63","Type":"ContainerDied","Data":"696732f5cd5524c151e92574e42f8d7098af380568a2bd18359e211c9afc7fb8"} Oct 30 13:14:32.245705 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:32.245489 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="696732f5cd5524c151e92574e42f8d7098af380568a2bd18359e211c9afc7fb8" Oct 30 13:14:32.261235 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:32.261215 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" Oct 30 13:14:32.367166 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:32.365435 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-credential-socket\") pod \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " Oct 30 13:14:32.367166 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:32.365493 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-workload-certs\") pod \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " Oct 30 13:14:32.367166 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:32.365624 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-workload-socket\") pod \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " Oct 30 13:14:32.367166 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:32.365674 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-istio-token\" (UniqueName: \"kubernetes.io/projected/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-istio-token\") pod \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " Oct 30 13:14:32.367166 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:32.365710 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-credential-socket" pod "a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63" (UID: "a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:14:32.367166 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:32.365733 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-istio-data\") pod \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " Oct 30 13:14:32.367166 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:32.365813 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-istio-podinfo\") pod \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " Oct 30 13:14:32.367166 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:32.365864 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-kube-api-access-7swjc\" (UniqueName: \"kubernetes.io/projected/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-kube-api-access-7swjc\") pod \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " Oct 30 13:14:32.367166 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:32.365973 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-workload-socket" pod "a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63" (UID: "a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:14:32.367166 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:32.366094 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-istiod-ca-cert\") pod \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " Oct 30 13:14:32.367166 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:32.366139 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-istio-envoy\") pod \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\" (UID: \"a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63\") " Oct 30 13:14:32.367166 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:32.366157 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-workload-certs" pod "a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63" (UID: "a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:14:32.367166 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:32.366832 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:14:32.367166 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:32.366857 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:14:32.367166 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:32.366871 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:14:32.368713 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:32.368675 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-istio-data" pod "a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63" (UID: "a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:14:32.368840 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:32.368761 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-istiod-ca-cert" pod "a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63" (UID: "a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:14:32.368926 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:32.368899 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-istio-podinfo" pod "a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63" (UID: "a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:14:32.371457 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:32.371397 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-istio-token" pod "a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63" (UID: "a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:14:32.378416 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:32.372371 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-istio-envoy" pod "a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63" (UID: "a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:14:32.378416 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:32.372434 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-kube-api-access-7swjc" pod "a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63" (UID: "a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63"). InnerVolumeSpecName "kube-api-access-7swjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:14:32.468184 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:32.468151 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:14:32.468184 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:32.468181 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:14:32.468396 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:32.468193 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7swjc\" (UniqueName: \"kubernetes.io/projected/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-kube-api-access-7swjc\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:14:32.468396 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:32.468203 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:14:32.468396 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:32.468212 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:14:32.468396 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:32.468221 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:14:33.252926 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:33.252893 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw" Oct 30 13:14:33.331678 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:33.331645 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw"] Oct 30 13:14:33.344450 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:33.344418 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw"] Oct 30 13:14:33.517454 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:33.517371 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63" path="/var/lib/kubelet/pods/a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63/volumes" Oct 30 13:14:37.353572 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:14:37.353534 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:14:41.165994 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:41.165960 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np"] Oct 30 13:14:41.166889 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:41.166870 4931 cpu_manager.go:414] "containerMap: removing container" podUID="a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63" containerName="istio-proxy" Oct 30 13:14:41.166971 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:41.166891 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63" containerName="istio-proxy" Oct 30 13:14:41.166971 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:41.166918 4931 cpu_manager.go:414] "containerMap: removing container" podUID="3037fde1-6ac2-4d15-8ecc-354dee56569c" containerName="revoke-keys" Oct 30 13:14:41.166971 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:41.166926 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3037fde1-6ac2-4d15-8ecc-354dee56569c" containerName="revoke-keys" Oct 30 13:14:41.167076 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:41.166982 4931 cpu_manager.go:414] "containerMap: removing container" podUID="fb4d115a-ce18-42fe-a26e-d28f0cbadd1e" containerName="main" Oct 30 13:14:41.167076 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:41.166990 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb4d115a-ce18-42fe-a26e-d28f0cbadd1e" containerName="main" Oct 30 13:14:41.167147 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:41.167135 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3e36d2e-bb23-46ba-bd2d-2f52fa9f4e63" containerName="istio-proxy" Oct 30 13:14:41.167178 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:41.167151 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="3037fde1-6ac2-4d15-8ecc-354dee56569c" containerName="revoke-keys" Oct 30 13:14:41.167178 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:41.167169 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb4d115a-ce18-42fe-a26e-d28f0cbadd1e" containerName="main" Oct 30 13:14:41.172663 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:41.172639 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np" Oct 30 13:14:41.193307 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:41.192521 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np"] Oct 30 13:14:41.201297 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:41.200530 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-4a8ee1-serving-cert\"" Oct 30 13:14:41.282425 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:41.282392 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c31718e-7be5-4591-b32b-bf9655c06c2a-metrics-certs\") pod \"maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np\" (UID: \"7c31718e-7be5-4591-b32b-bf9655c06c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np" Oct 30 13:14:41.282593 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:41.282547 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c31718e-7be5-4591-b32b-bf9655c06c2a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np\" (UID: \"7c31718e-7be5-4591-b32b-bf9655c06c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np" Oct 30 13:14:41.282700 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:41.282681 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c31718e-7be5-4591-b32b-bf9655c06c2a-maas-api-tls\") pod \"maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np\" (UID: \"7c31718e-7be5-4591-b32b-bf9655c06c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np" Oct 30 13:14:41.282761 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:41.282746 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94vj8\" (UniqueName: \"kubernetes.io/projected/7c31718e-7be5-4591-b32b-bf9655c06c2a-kube-api-access-94vj8\") pod \"maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np\" (UID: \"7c31718e-7be5-4591-b32b-bf9655c06c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np" Oct 30 13:14:41.387953 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:41.387911 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c31718e-7be5-4591-b32b-bf9655c06c2a-maas-api-tls\") pod \"maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np\" (UID: \"7c31718e-7be5-4591-b32b-bf9655c06c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np" Oct 30 13:14:41.388237 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:41.388219 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94vj8\" (UniqueName: \"kubernetes.io/projected/7c31718e-7be5-4591-b32b-bf9655c06c2a-kube-api-access-94vj8\") pod \"maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np\" (UID: \"7c31718e-7be5-4591-b32b-bf9655c06c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np" Oct 30 13:14:41.388462 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:41.388445 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c31718e-7be5-4591-b32b-bf9655c06c2a-metrics-certs\") pod \"maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np\" (UID: \"7c31718e-7be5-4591-b32b-bf9655c06c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np" Oct 30 13:14:41.388665 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:41.388649 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c31718e-7be5-4591-b32b-bf9655c06c2a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np\" (UID: \"7c31718e-7be5-4591-b32b-bf9655c06c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np" Oct 30 13:14:41.394666 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:41.394635 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c31718e-7be5-4591-b32b-bf9655c06c2a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np\" (UID: \"7c31718e-7be5-4591-b32b-bf9655c06c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np" Oct 30 13:14:41.401764 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:41.398884 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c31718e-7be5-4591-b32b-bf9655c06c2a-metrics-certs\") pod \"maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np\" (UID: \"7c31718e-7be5-4591-b32b-bf9655c06c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np" Oct 30 13:14:41.408985 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:41.408346 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c31718e-7be5-4591-b32b-bf9655c06c2a-maas-api-tls\") pod \"maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np\" (UID: \"7c31718e-7be5-4591-b32b-bf9655c06c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np" Oct 30 13:14:41.427917 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:41.427843 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94vj8\" (UniqueName: \"kubernetes.io/projected/7c31718e-7be5-4591-b32b-bf9655c06c2a-kube-api-access-94vj8\") pod \"maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np\" (UID: \"7c31718e-7be5-4591-b32b-bf9655c06c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np" Oct 30 13:14:41.493864 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:41.493786 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np" Oct 30 13:14:41.864444 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:41.864411 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np"] Oct 30 13:14:41.864848 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:14:41.864815 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c31718e_7be5_4591_b32b_bf9655c06c2a.slice/crio-84d2952ea9b3bd12b1ff09032f61271fbc3a679a5349c58c70cbc47d92bd5202 WatchSource:0}: Error finding container 84d2952ea9b3bd12b1ff09032f61271fbc3a679a5349c58c70cbc47d92bd5202: Status 404 returned error can't find the container with id 84d2952ea9b3bd12b1ff09032f61271fbc3a679a5349c58c70cbc47d92bd5202 Oct 30 13:14:41.866099 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:41.866080 4931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:14:42.455018 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:42.452982 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np" event={"ID":"7c31718e-7be5-4591-b32b-bf9655c06c2a","Type":"ContainerStarted","Data":"64a185a6f3369676255fcdca7e8bcb76ea453690e0a092f83616a43727af1fb8"} Oct 30 13:14:42.455018 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:42.453051 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np" event={"ID":"7c31718e-7be5-4591-b32b-bf9655c06c2a","Type":"ContainerStarted","Data":"84d2952ea9b3bd12b1ff09032f61271fbc3a679a5349c58c70cbc47d92bd5202"} Oct 30 13:14:42.455018 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:42.454174 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np" Oct 30 13:14:42.515413 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:42.515360 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np" podStartSLOduration=1.177362444 podStartE2EDuration="1.515344796s" totalImagesPullingTime="337.982352ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:42.513084148 +0000 UTC m=+1781.116582596" watchObservedRunningTime="2026-10-30 13:14:42.515344796 +0000 UTC m=+1781.118843242" Oct 30 13:14:44.609135 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:44.609096 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b9e8e-d75gd"] Oct 30 13:14:44.614036 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:44.614019 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b9e8e-d75gd" Oct 30 13:14:44.666002 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:44.665967 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b9e8e-d75gd"] Oct 30 13:14:44.698096 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:44.698063 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:14:44.698338 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:44.698153 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:14:44.701209 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:44.701179 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29227526-2318-4d64-8423-bb63666f80c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5b9e8e-d75gd\" (UID: \"29227526-2318-4d64-8423-bb63666f80c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b9e8e-d75gd" Oct 30 13:14:44.701401 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:44.701383 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29227526-2318-4d64-8423-bb63666f80c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5b9e8e-d75gd\" (UID: \"29227526-2318-4d64-8423-bb63666f80c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b9e8e-d75gd" Oct 30 13:14:44.701539 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:44.701514 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2blwt\" (UniqueName: \"kubernetes.io/projected/29227526-2318-4d64-8423-bb63666f80c7-kube-api-access-2blwt\") pod \"maas-api-revoke-keys-e2e-api-a-5b9e8e-d75gd\" (UID: \"29227526-2318-4d64-8423-bb63666f80c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b9e8e-d75gd" Oct 30 13:14:44.802533 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:44.802499 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29227526-2318-4d64-8423-bb63666f80c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5b9e8e-d75gd\" (UID: \"29227526-2318-4d64-8423-bb63666f80c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b9e8e-d75gd" Oct 30 13:14:44.802703 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:44.802620 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2blwt\" (UniqueName: \"kubernetes.io/projected/29227526-2318-4d64-8423-bb63666f80c7-kube-api-access-2blwt\") pod \"maas-api-revoke-keys-e2e-api-a-5b9e8e-d75gd\" (UID: \"29227526-2318-4d64-8423-bb63666f80c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b9e8e-d75gd" Oct 30 13:14:44.802790 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:44.802773 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29227526-2318-4d64-8423-bb63666f80c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5b9e8e-d75gd\" (UID: \"29227526-2318-4d64-8423-bb63666f80c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b9e8e-d75gd" Oct 30 13:14:44.803248 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:44.803224 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29227526-2318-4d64-8423-bb63666f80c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5b9e8e-d75gd\" (UID: \"29227526-2318-4d64-8423-bb63666f80c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b9e8e-d75gd" Oct 30 13:14:44.805345 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:44.805324 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29227526-2318-4d64-8423-bb63666f80c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5b9e8e-d75gd\" (UID: \"29227526-2318-4d64-8423-bb63666f80c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b9e8e-d75gd" Oct 30 13:14:44.828718 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:44.828692 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2blwt\" (UniqueName: \"kubernetes.io/projected/29227526-2318-4d64-8423-bb63666f80c7-kube-api-access-2blwt\") pod \"maas-api-revoke-keys-e2e-api-a-5b9e8e-d75gd\" (UID: \"29227526-2318-4d64-8423-bb63666f80c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b9e8e-d75gd" Oct 30 13:14:44.928711 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:44.928620 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b9e8e-d75gd" Oct 30 13:14:45.273613 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:14:45.273580 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29227526_2318_4d64_8423_bb63666f80c7.slice/crio-0de01bf2c3acc5789774aa2ffb6c1256386297e0e7c84d3c48291269bb5788bd WatchSource:0}: Error finding container 0de01bf2c3acc5789774aa2ffb6c1256386297e0e7c84d3c48291269bb5788bd: Status 404 returned error can't find the container with id 0de01bf2c3acc5789774aa2ffb6c1256386297e0e7c84d3c48291269bb5788bd Oct 30 13:14:45.273763 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:45.273743 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b9e8e-d75gd"] Oct 30 13:14:45.502902 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:45.502869 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b9e8e-d75gd" event={"ID":"29227526-2318-4d64-8423-bb63666f80c7","Type":"ContainerStarted","Data":"0de01bf2c3acc5789774aa2ffb6c1256386297e0e7c84d3c48291269bb5788bd"} Oct 30 13:14:46.252600 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:46.252555 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q"] Oct 30 13:14:46.253039 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:46.252834 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" podUID="20a13ccd-7ec5-4401-b316-77e26b4c6f7b" containerName="istio-proxy" containerID="cri-o://7ad486931063396f67048d748c0088b6fb03845f0e8b3f3416573523a613475b" gracePeriod=30 Oct 30 13:14:46.531359 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:46.530080 4931 generic.go:356] "Generic (PLEG): container finished" podID="29227526-2318-4d64-8423-bb63666f80c7" containerID="c1f39d7956890aefc615831680faeaf153791c9868a87e28e3e3e00585b304b9" exitCode=0 Oct 30 13:14:46.531359 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:46.530322 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b9e8e-d75gd" event={"ID":"29227526-2318-4d64-8423-bb63666f80c7","Type":"ContainerDied","Data":"c1f39d7956890aefc615831680faeaf153791c9868a87e28e3e3e00585b304b9"} Oct 30 13:14:47.744672 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:47.744643 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b9e8e-d75gd" Oct 30 13:14:47.853830 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:47.853744 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29227526-2318-4d64-8423-bb63666f80c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29227526-2318-4d64-8423-bb63666f80c7-maas-api-openshift-ca\") pod \"29227526-2318-4d64-8423-bb63666f80c7\" (UID: \"29227526-2318-4d64-8423-bb63666f80c7\") " Oct 30 13:14:47.854442 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:47.854416 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29227526-2318-4d64-8423-bb63666f80c7-kube-api-access-2blwt\" (UniqueName: \"kubernetes.io/projected/29227526-2318-4d64-8423-bb63666f80c7-kube-api-access-2blwt\") pod \"29227526-2318-4d64-8423-bb63666f80c7\" (UID: \"29227526-2318-4d64-8423-bb63666f80c7\") " Oct 30 13:14:47.854749 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:47.854727 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29227526-2318-4d64-8423-bb63666f80c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29227526-2318-4d64-8423-bb63666f80c7-maas-api-xks-ca\") pod \"29227526-2318-4d64-8423-bb63666f80c7\" (UID: \"29227526-2318-4d64-8423-bb63666f80c7\") " Oct 30 13:14:47.855744 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:47.854220 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29227526-2318-4d64-8423-bb63666f80c7-maas-api-openshift-ca" pod "29227526-2318-4d64-8423-bb63666f80c7" (UID: "29227526-2318-4d64-8423-bb63666f80c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:14:47.858976 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:47.858937 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29227526-2318-4d64-8423-bb63666f80c7-kube-api-access-2blwt" pod "29227526-2318-4d64-8423-bb63666f80c7" (UID: "29227526-2318-4d64-8423-bb63666f80c7"). InnerVolumeSpecName "kube-api-access-2blwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:14:47.859180 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:47.859153 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29227526-2318-4d64-8423-bb63666f80c7-maas-api-xks-ca" pod "29227526-2318-4d64-8423-bb63666f80c7" (UID: "29227526-2318-4d64-8423-bb63666f80c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:14:47.957787 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:47.957748 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2blwt\" (UniqueName: \"kubernetes.io/projected/29227526-2318-4d64-8423-bb63666f80c7-kube-api-access-2blwt\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:14:47.957787 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:47.957782 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29227526-2318-4d64-8423-bb63666f80c7-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:14:47.957787 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:47.957792 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29227526-2318-4d64-8423-bb63666f80c7-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:14:48.041550 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:48.041514 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70ba7ee1-v2nlk"] Oct 30 13:14:48.042577 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:48.042552 4931 cpu_manager.go:414] "containerMap: removing container" podUID="29227526-2318-4d64-8423-bb63666f80c7" containerName="revoke-keys" Oct 30 13:14:48.042690 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:48.042579 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29227526-2318-4d64-8423-bb63666f80c7" containerName="revoke-keys" Oct 30 13:14:48.042758 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:48.042742 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="29227526-2318-4d64-8423-bb63666f80c7" containerName="revoke-keys" Oct 30 13:14:48.047118 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:48.047097 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ba7ee1-v2nlk" Oct 30 13:14:48.070881 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:48.070845 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70ba7ee1-v2nlk"] Oct 30 13:14:48.162046 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:48.161947 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc5c16ca-a37d-41c0-8bd4-246a23d03189-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70ba7ee1-v2nlk\" (UID: \"dc5c16ca-a37d-41c0-8bd4-246a23d03189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ba7ee1-v2nlk" Oct 30 13:14:48.162218 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:48.162088 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8zff\" (UniqueName: \"kubernetes.io/projected/dc5c16ca-a37d-41c0-8bd4-246a23d03189-kube-api-access-j8zff\") pod \"maas-api-revoke-sub-70ba7ee1-v2nlk\" (UID: \"dc5c16ca-a37d-41c0-8bd4-246a23d03189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ba7ee1-v2nlk" Oct 30 13:14:48.162218 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:48.162192 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc5c16ca-a37d-41c0-8bd4-246a23d03189-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70ba7ee1-v2nlk\" (UID: \"dc5c16ca-a37d-41c0-8bd4-246a23d03189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ba7ee1-v2nlk" Oct 30 13:14:48.263977 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:48.263937 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc5c16ca-a37d-41c0-8bd4-246a23d03189-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70ba7ee1-v2nlk\" (UID: \"dc5c16ca-a37d-41c0-8bd4-246a23d03189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ba7ee1-v2nlk" Oct 30 13:14:48.264184 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:48.264018 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8zff\" (UniqueName: \"kubernetes.io/projected/dc5c16ca-a37d-41c0-8bd4-246a23d03189-kube-api-access-j8zff\") pod \"maas-api-revoke-sub-70ba7ee1-v2nlk\" (UID: \"dc5c16ca-a37d-41c0-8bd4-246a23d03189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ba7ee1-v2nlk" Oct 30 13:14:48.264184 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:48.264097 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc5c16ca-a37d-41c0-8bd4-246a23d03189-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70ba7ee1-v2nlk\" (UID: \"dc5c16ca-a37d-41c0-8bd4-246a23d03189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ba7ee1-v2nlk" Oct 30 13:14:48.264866 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:48.264836 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc5c16ca-a37d-41c0-8bd4-246a23d03189-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70ba7ee1-v2nlk\" (UID: \"dc5c16ca-a37d-41c0-8bd4-246a23d03189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ba7ee1-v2nlk" Oct 30 13:14:48.268984 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:48.268957 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc5c16ca-a37d-41c0-8bd4-246a23d03189-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70ba7ee1-v2nlk\" (UID: \"dc5c16ca-a37d-41c0-8bd4-246a23d03189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ba7ee1-v2nlk" Oct 30 13:14:48.294651 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:48.294622 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8zff\" (UniqueName: \"kubernetes.io/projected/dc5c16ca-a37d-41c0-8bd4-246a23d03189-kube-api-access-j8zff\") pod \"maas-api-revoke-sub-70ba7ee1-v2nlk\" (UID: \"dc5c16ca-a37d-41c0-8bd4-246a23d03189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ba7ee1-v2nlk" Oct 30 13:14:48.370353 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:48.370299 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ba7ee1-v2nlk" Oct 30 13:14:48.552165 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:48.552138 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b9e8e-d75gd" Oct 30 13:14:48.552165 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:48.552156 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b9e8e-d75gd" event={"ID":"29227526-2318-4d64-8423-bb63666f80c7","Type":"ContainerDied","Data":"0de01bf2c3acc5789774aa2ffb6c1256386297e0e7c84d3c48291269bb5788bd"} Oct 30 13:14:48.552509 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:48.552206 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0de01bf2c3acc5789774aa2ffb6c1256386297e0e7c84d3c48291269bb5788bd" Oct 30 13:14:48.702642 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:48.702599 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70ba7ee1-v2nlk"] Oct 30 13:14:49.500026 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:49.499977 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np" Oct 30 13:14:49.561606 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:49.561569 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ba7ee1-v2nlk" event={"ID":"dc5c16ca-a37d-41c0-8bd4-246a23d03189","Type":"ContainerStarted","Data":"4539c9c0d36d2a7a5b72d2d661becd7739d2c7cba9c8f5e3ecdb0cff6c995273"} Oct 30 13:14:49.561880 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:49.561631 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ba7ee1-v2nlk" event={"ID":"dc5c16ca-a37d-41c0-8bd4-246a23d03189","Type":"ContainerStarted","Data":"bc021beaa9e5ce8983c2bba8073b1078ac69ea5aa98445d692f41160aae6b54e"} Oct 30 13:14:49.563172 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:49.563149 4931 generic.go:356] "Generic (PLEG): container finished" podID="9ef8a128-1b22-4ef2-81ed-0a1b494393c3" containerID="4e1eeee4d82f56f22f06c4a0c2ed45a49a1fb852be0f2d410ed0f0ed98784c6e" exitCode=0 Oct 30 13:14:49.563301 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:49.563186 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" event={"ID":"9ef8a128-1b22-4ef2-81ed-0a1b494393c3","Type":"ContainerDied","Data":"4e1eeee4d82f56f22f06c4a0c2ed45a49a1fb852be0f2d410ed0f0ed98784c6e"} Oct 30 13:14:49.563550 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:49.563537 4931 scope.go:122] "RemoveContainer" containerID="4e1eeee4d82f56f22f06c4a0c2ed45a49a1fb852be0f2d410ed0f0ed98784c6e" Oct 30 13:14:49.565162 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:49.565133 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:14:49.565236 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:49.565201 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:14:49.565236 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:49.565231 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:14:49.786767 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:49.786710 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ba7ee1-v2nlk" podStartSLOduration=1.786687414 podStartE2EDuration="1.786687414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:49.782815724 +0000 UTC m=+1788.386314171" watchObservedRunningTime="2026-10-30 13:14:49.786687414 +0000 UTC m=+1788.390185864" Oct 30 13:14:50.574163 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:50.574127 4931 generic.go:356] "Generic (PLEG): container finished" podID="dc5c16ca-a37d-41c0-8bd4-246a23d03189" containerID="4539c9c0d36d2a7a5b72d2d661becd7739d2c7cba9c8f5e3ecdb0cff6c995273" exitCode=0 Oct 30 13:14:50.574707 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:50.574222 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ba7ee1-v2nlk" event={"ID":"dc5c16ca-a37d-41c0-8bd4-246a23d03189","Type":"ContainerDied","Data":"4539c9c0d36d2a7a5b72d2d661becd7739d2c7cba9c8f5e3ecdb0cff6c995273"} Oct 30 13:14:50.576416 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:50.576391 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" event={"ID":"9ef8a128-1b22-4ef2-81ed-0a1b494393c3","Type":"ContainerStarted","Data":"528bea8ad0c34002c1538d2e3b57ada6dd8a24fb147c10ef24ae8c52b18470e4"} Oct 30 13:14:50.926045 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:50.925950 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:14:50.926498 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:50.926470 4931 patch_prober.go:29] interesting pod/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 13:14:50.926651 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:50.926523 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" podUID="9ef8a128-1b22-4ef2-81ed-0a1b494393c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 13:14:51.586136 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.586106 4931 generic.go:356] "Generic (PLEG): container finished" podID="20a13ccd-7ec5-4401-b316-77e26b4c6f7b" containerID="7ad486931063396f67048d748c0088b6fb03845f0e8b3f3416573523a613475b" exitCode=0 Oct 30 13:14:51.586604 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.586182 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" event={"ID":"20a13ccd-7ec5-4401-b316-77e26b4c6f7b","Type":"ContainerDied","Data":"7ad486931063396f67048d748c0088b6fb03845f0e8b3f3416573523a613475b"} Oct 30 13:14:51.586604 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.586239 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" event={"ID":"20a13ccd-7ec5-4401-b316-77e26b4c6f7b","Type":"ContainerDied","Data":"7ddcab40e3e84bc1efde1ac2887a918a05f196404f11e0af83463f688f7b13a0"} Oct 30 13:14:51.586604 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.586256 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ddcab40e3e84bc1efde1ac2887a918a05f196404f11e0af83463f688f7b13a0" Oct 30 13:14:51.599332 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.599308 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" Oct 30 13:14:51.615643 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.615609 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-workload-socket\") pod \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " Oct 30 13:14:51.615854 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.615667 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-istio-data\") pod \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " Oct 30 13:14:51.615854 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.615717 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-kube-api-access-7pr5t\" (UniqueName: \"kubernetes.io/projected/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-kube-api-access-7pr5t\") pod \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " Oct 30 13:14:51.615854 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.615764 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-istiod-ca-cert\") pod \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " Oct 30 13:14:51.616034 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.615856 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-istio-token\" (UniqueName: \"kubernetes.io/projected/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-istio-token\") pod \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " Oct 30 13:14:51.616034 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.615918 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-istio-podinfo\") pod \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " Oct 30 13:14:51.616034 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.615957 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-credential-socket\") pod \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " Oct 30 13:14:51.616034 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.616000 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-istio-envoy\") pod \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " Oct 30 13:14:51.616242 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.615850 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-workload-socket" pod "20a13ccd-7ec5-4401-b316-77e26b4c6f7b" (UID: "20a13ccd-7ec5-4401-b316-77e26b4c6f7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:14:51.616242 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.616065 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-workload-certs\") pod \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\" (UID: \"20a13ccd-7ec5-4401-b316-77e26b4c6f7b\") " Oct 30 13:14:51.616514 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.616487 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-istio-data" pod "20a13ccd-7ec5-4401-b316-77e26b4c6f7b" (UID: "20a13ccd-7ec5-4401-b316-77e26b4c6f7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:14:51.616689 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.616613 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-workload-certs" pod "20a13ccd-7ec5-4401-b316-77e26b4c6f7b" (UID: "20a13ccd-7ec5-4401-b316-77e26b4c6f7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:14:51.618782 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.616975 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-credential-socket" pod "20a13ccd-7ec5-4401-b316-77e26b4c6f7b" (UID: "20a13ccd-7ec5-4401-b316-77e26b4c6f7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:14:51.618782 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.617829 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-istiod-ca-cert" pod "20a13ccd-7ec5-4401-b316-77e26b4c6f7b" (UID: "20a13ccd-7ec5-4401-b316-77e26b4c6f7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:14:51.618956 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.618831 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:14:51.618956 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.618855 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:14:51.618956 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.618869 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:14:51.618956 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.618881 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:14:51.618956 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.618893 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:14:51.619551 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.619529 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-kube-api-access-7pr5t" pod "20a13ccd-7ec5-4401-b316-77e26b4c6f7b" (UID: "20a13ccd-7ec5-4401-b316-77e26b4c6f7b"). InnerVolumeSpecName "kube-api-access-7pr5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:14:51.623759 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.623718 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-istio-envoy" pod "20a13ccd-7ec5-4401-b316-77e26b4c6f7b" (UID: "20a13ccd-7ec5-4401-b316-77e26b4c6f7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:14:51.625427 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.625402 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-istio-podinfo" pod "20a13ccd-7ec5-4401-b316-77e26b4c6f7b" (UID: "20a13ccd-7ec5-4401-b316-77e26b4c6f7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:14:51.626878 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.626728 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-istio-token" pod "20a13ccd-7ec5-4401-b316-77e26b4c6f7b" (UID: "20a13ccd-7ec5-4401-b316-77e26b4c6f7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:14:51.722477 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.722441 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pr5t\" (UniqueName: \"kubernetes.io/projected/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-kube-api-access-7pr5t\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:14:51.722625 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.722480 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:14:51.722625 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.722496 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:14:51.722625 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.722508 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20a13ccd-7ec5-4401-b316-77e26b4c6f7b-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:14:51.831464 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.831434 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ba7ee1-v2nlk" Oct 30 13:14:51.926591 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.926550 4931 patch_prober.go:29] interesting pod/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 13:14:51.926790 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.926612 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc5c16ca-a37d-41c0-8bd4-246a23d03189-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc5c16ca-a37d-41c0-8bd4-246a23d03189-maas-api-openshift-ca\") pod \"dc5c16ca-a37d-41c0-8bd4-246a23d03189\" (UID: \"dc5c16ca-a37d-41c0-8bd4-246a23d03189\") " Oct 30 13:14:51.926790 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.926619 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" podUID="9ef8a128-1b22-4ef2-81ed-0a1b494393c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 13:14:51.926790 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.926669 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc5c16ca-a37d-41c0-8bd4-246a23d03189-kube-api-access-j8zff\" (UniqueName: \"kubernetes.io/projected/dc5c16ca-a37d-41c0-8bd4-246a23d03189-kube-api-access-j8zff\") pod \"dc5c16ca-a37d-41c0-8bd4-246a23d03189\" (UID: \"dc5c16ca-a37d-41c0-8bd4-246a23d03189\") " Oct 30 13:14:51.926976 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.926816 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc5c16ca-a37d-41c0-8bd4-246a23d03189-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc5c16ca-a37d-41c0-8bd4-246a23d03189-maas-api-xks-ca\") pod \"dc5c16ca-a37d-41c0-8bd4-246a23d03189\" (UID: \"dc5c16ca-a37d-41c0-8bd4-246a23d03189\") " Oct 30 13:14:51.927829 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.927802 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc5c16ca-a37d-41c0-8bd4-246a23d03189-maas-api-openshift-ca" pod "dc5c16ca-a37d-41c0-8bd4-246a23d03189" (UID: "dc5c16ca-a37d-41c0-8bd4-246a23d03189"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:14:51.929988 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.929958 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc5c16ca-a37d-41c0-8bd4-246a23d03189-maas-api-xks-ca" pod "dc5c16ca-a37d-41c0-8bd4-246a23d03189" (UID: "dc5c16ca-a37d-41c0-8bd4-246a23d03189"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:14:51.930133 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:51.930087 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc5c16ca-a37d-41c0-8bd4-246a23d03189-kube-api-access-j8zff" pod "dc5c16ca-a37d-41c0-8bd4-246a23d03189" (UID: "dc5c16ca-a37d-41c0-8bd4-246a23d03189"). InnerVolumeSpecName "kube-api-access-j8zff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:14:52.029578 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:52.029544 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc5c16ca-a37d-41c0-8bd4-246a23d03189-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:14:52.029578 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:52.029574 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8zff\" (UniqueName: \"kubernetes.io/projected/dc5c16ca-a37d-41c0-8bd4-246a23d03189-kube-api-access-j8zff\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:14:52.029578 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:52.029584 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc5c16ca-a37d-41c0-8bd4-246a23d03189-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:14:52.596567 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:52.596481 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ba7ee1-v2nlk" Oct 30 13:14:52.596567 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:52.596510 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ba7ee1-v2nlk" event={"ID":"dc5c16ca-a37d-41c0-8bd4-246a23d03189","Type":"ContainerDied","Data":"bc021beaa9e5ce8983c2bba8073b1078ac69ea5aa98445d692f41160aae6b54e"} Oct 30 13:14:52.596567 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:52.596561 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc021beaa9e5ce8983c2bba8073b1078ac69ea5aa98445d692f41160aae6b54e" Oct 30 13:14:52.597773 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:52.596718 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q" Oct 30 13:14:52.678625 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:52.676484 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q"] Oct 30 13:14:52.683770 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:52.683745 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q"] Oct 30 13:14:52.926032 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:52.925937 4931 patch_prober.go:29] interesting pod/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 13:14:52.926032 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:52.926004 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" podUID="9ef8a128-1b22-4ef2-81ed-0a1b494393c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 13:14:53.516252 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:53.516218 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20a13ccd-7ec5-4401-b316-77e26b4c6f7b" path="/var/lib/kubelet/pods/20a13ccd-7ec5-4401-b316-77e26b4c6f7b/volumes" Oct 30 13:14:53.926919 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:53.926881 4931 patch_prober.go:29] interesting pod/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 13:14:53.927358 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:53.926945 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" podUID="9ef8a128-1b22-4ef2-81ed-0a1b494393c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 13:14:54.926839 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:54.926801 4931 patch_prober.go:29] interesting pod/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 13:14:54.927016 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:54.926868 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" podUID="9ef8a128-1b22-4ef2-81ed-0a1b494393c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 13:14:55.003658 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:55.003620 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3160a-ww8z2"] Oct 30 13:14:55.016642 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:55.016603 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3160a-ww8z2"] Oct 30 13:14:55.523249 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:55.523200 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3242035b-ebb3-45dc-9af0-c0ce730be9ca" path="/var/lib/kubelet/pods/3242035b-ebb3-45dc-9af0-c0ce730be9ca/volumes" Oct 30 13:14:55.997647 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:55.997605 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:14:55.998204 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:55.998176 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:14:55.999559 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:14:55.999536 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:15:00.176891 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.176854 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj"] Oct 30 13:15:00.177729 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.177708 4931 cpu_manager.go:414] "containerMap: removing container" podUID="20a13ccd-7ec5-4401-b316-77e26b4c6f7b" containerName="istio-proxy" Oct 30 13:15:00.177729 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.177730 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20a13ccd-7ec5-4401-b316-77e26b4c6f7b" containerName="istio-proxy" Oct 30 13:15:00.177933 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.177803 4931 cpu_manager.go:414] "containerMap: removing container" podUID="dc5c16ca-a37d-41c0-8bd4-246a23d03189" containerName="revoke-keys" Oct 30 13:15:00.177933 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.177813 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc5c16ca-a37d-41c0-8bd4-246a23d03189" containerName="revoke-keys" Oct 30 13:15:00.178043 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.177977 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc5c16ca-a37d-41c0-8bd4-246a23d03189" containerName="revoke-keys" Oct 30 13:15:00.178043 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.178000 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="20a13ccd-7ec5-4401-b316-77e26b4c6f7b" containerName="istio-proxy" Oct 30 13:15:00.181812 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.181793 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj" Oct 30 13:15:00.208036 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.208005 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh"] Oct 30 13:15:00.215925 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.215475 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:15:00.221773 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.221743 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh" Oct 30 13:15:00.230212 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.230096 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9btl9\" (UniqueName: \"kubernetes.io/projected/01a78c34-1e58-48f3-8660-56d6b19d8847-kube-api-access-9btl9\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh\" (UID: \"01a78c34-1e58-48f3-8660-56d6b19d8847\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh" Oct 30 13:15:00.230212 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.230196 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea3c064a-c226-4eec-bd7e-d42fb03965ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj\" (UID: \"ea3c064a-c226-4eec-bd7e-d42fb03965ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj" Oct 30 13:15:00.230449 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.230394 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a78c34-1e58-48f3-8660-56d6b19d8847-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh\" (UID: \"01a78c34-1e58-48f3-8660-56d6b19d8847\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh" Oct 30 13:15:00.230568 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.230548 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh7v9\" (UniqueName: \"kubernetes.io/projected/ea3c064a-c226-4eec-bd7e-d42fb03965ad-kube-api-access-vh7v9\") pod \"maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj\" (UID: \"ea3c064a-c226-4eec-bd7e-d42fb03965ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj" Oct 30 13:15:00.251563 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.251536 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889435-q4q4t"] Oct 30 13:15:00.258733 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.256837 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj"] Oct 30 13:15:00.258733 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.256951 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889435-q4q4t" Oct 30 13:15:00.265359 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.265331 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh"] Oct 30 13:15:00.276186 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.276163 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj"] Oct 30 13:15:00.280974 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.280953 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj" Oct 30 13:15:00.282048 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.282028 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2"] Oct 30 13:15:00.287463 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.287442 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2" Oct 30 13:15:00.287570 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.287543 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889435-q4q4t"] Oct 30 13:15:00.330834 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.330796 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5b9e8e-29889435-6cs4n"] Oct 30 13:15:00.332206 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.332171 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh7v9\" (UniqueName: \"kubernetes.io/projected/ea3c064a-c226-4eec-bd7e-d42fb03965ad-kube-api-access-vh7v9\") pod \"maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj\" (UID: \"ea3c064a-c226-4eec-bd7e-d42fb03965ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj" Oct 30 13:15:00.332356 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.332333 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9btl9\" (UniqueName: \"kubernetes.io/projected/01a78c34-1e58-48f3-8660-56d6b19d8847-kube-api-access-9btl9\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh\" (UID: \"01a78c34-1e58-48f3-8660-56d6b19d8847\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh" Oct 30 13:15:00.332436 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.332420 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea3c064a-c226-4eec-bd7e-d42fb03965ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj\" (UID: \"ea3c064a-c226-4eec-bd7e-d42fb03965ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj" Oct 30 13:15:00.332551 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.332534 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a78c34-1e58-48f3-8660-56d6b19d8847-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh\" (UID: \"01a78c34-1e58-48f3-8660-56d6b19d8847\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh" Oct 30 13:15:00.333478 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.333420 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a78c34-1e58-48f3-8660-56d6b19d8847-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh\" (UID: \"01a78c34-1e58-48f3-8660-56d6b19d8847\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh" Oct 30 13:15:00.334617 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.334549 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea3c064a-c226-4eec-bd7e-d42fb03965ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj\" (UID: \"ea3c064a-c226-4eec-bd7e-d42fb03965ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj" Oct 30 13:15:00.344850 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.344827 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj"] Oct 30 13:15:00.344934 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.344871 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2"] Oct 30 13:15:00.344986 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.344965 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5b9e8e-29889435-6cs4n" Oct 30 13:15:00.390611 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.389522 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9btl9\" (UniqueName: \"kubernetes.io/projected/01a78c34-1e58-48f3-8660-56d6b19d8847-kube-api-access-9btl9\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh\" (UID: \"01a78c34-1e58-48f3-8660-56d6b19d8847\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh" Oct 30 13:15:00.390611 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.389704 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5b9e8e-29889435-6cs4n"] Oct 30 13:15:00.404551 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.404511 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4a8ee1-29889435-5w88z"] Oct 30 13:15:00.411329 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.411252 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh7v9\" (UniqueName: \"kubernetes.io/projected/ea3c064a-c226-4eec-bd7e-d42fb03965ad-kube-api-access-vh7v9\") pod \"maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj\" (UID: \"ea3c064a-c226-4eec-bd7e-d42fb03965ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj" Oct 30 13:15:00.418895 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.418875 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4a8ee1-29889435-5w88z" Oct 30 13:15:00.425332 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.423966 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv"] Oct 30 13:15:00.428504 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.428438 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv" Oct 30 13:15:00.434526 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.434495 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3bc7915-8ed0-47c5-a693-15db3d153fe8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj\" (UID: \"c3bc7915-8ed0-47c5-a693-15db3d153fe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj" Oct 30 13:15:00.434695 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.434664 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/795f8222-5020-4ef8-be21-49fbb8b178e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889435-q4q4t\" (UID: \"795f8222-5020-4ef8-be21-49fbb8b178e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889435-q4q4t" Oct 30 13:15:00.434949 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.434923 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a5f8729-df67-4e02-b7c5-f2250efa9678-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2\" (UID: \"2a5f8729-df67-4e02-b7c5-f2250efa9678\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2" Oct 30 13:15:00.435036 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.434993 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpxxf\" (UniqueName: \"kubernetes.io/projected/c3bc7915-8ed0-47c5-a693-15db3d153fe8-kube-api-access-tpxxf\") pod \"maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj\" (UID: \"c3bc7915-8ed0-47c5-a693-15db3d153fe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj" Oct 30 13:15:00.435179 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.435131 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktxdj\" (UniqueName: \"kubernetes.io/projected/795f8222-5020-4ef8-be21-49fbb8b178e8-kube-api-access-ktxdj\") pod \"maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889435-q4q4t\" (UID: \"795f8222-5020-4ef8-be21-49fbb8b178e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889435-q4q4t" Oct 30 13:15:00.435284 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.435185 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ggzk\" (UniqueName: \"kubernetes.io/projected/2a5f8729-df67-4e02-b7c5-f2250efa9678-kube-api-access-2ggzk\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2\" (UID: \"2a5f8729-df67-4e02-b7c5-f2250efa9678\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2" Oct 30 13:15:00.438572 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.438551 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4a8ee1-29889435-5w88z"] Oct 30 13:15:00.444135 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.444111 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31e10b-29889435-tt6md"] Oct 30 13:15:00.451524 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.451504 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31e10b-29889435-tt6md" Oct 30 13:15:00.478834 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.478805 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv"] Oct 30 13:15:00.495917 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.495886 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889435-h2hfn"] Oct 30 13:15:00.496150 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.496121 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj" Oct 30 13:15:00.503056 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.503038 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-h2hfn" Oct 30 13:15:00.510214 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.510180 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31e10b-29889435-tt6md"] Oct 30 13:15:00.521554 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.521526 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-14e8a4-29889435-cldf2"] Oct 30 13:15:00.529039 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.529014 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-14e8a4-29889435-cldf2" Oct 30 13:15:00.541828 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.540320 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpxxf\" (UniqueName: \"kubernetes.io/projected/c3bc7915-8ed0-47c5-a693-15db3d153fe8-kube-api-access-tpxxf\") pod \"maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj\" (UID: \"c3bc7915-8ed0-47c5-a693-15db3d153fe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj" Oct 30 13:15:00.541828 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.540376 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khkhv\" (UniqueName: \"kubernetes.io/projected/f1814684-56ff-4b44-8328-29cbe0497bf2-kube-api-access-khkhv\") pod \"maas-api-key-cleanup-e2e-api-b-31e10b-29889435-tt6md\" (UID: \"f1814684-56ff-4b44-8328-29cbe0497bf2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31e10b-29889435-tt6md" Oct 30 13:15:00.541828 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.540445 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d882efb1-cd4f-45ce-b829-4795003e31b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-14e8a4-29889435-cldf2\" (UID: \"d882efb1-cd4f-45ce-b829-4795003e31b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-14e8a4-29889435-cldf2" Oct 30 13:15:00.541828 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.540476 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktxdj\" (UniqueName: \"kubernetes.io/projected/795f8222-5020-4ef8-be21-49fbb8b178e8-kube-api-access-ktxdj\") pod \"maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889435-q4q4t\" (UID: \"795f8222-5020-4ef8-be21-49fbb8b178e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889435-q4q4t" Oct 30 13:15:00.541828 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.540521 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ggzk\" (UniqueName: \"kubernetes.io/projected/2a5f8729-df67-4e02-b7c5-f2250efa9678-kube-api-access-2ggzk\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2\" (UID: \"2a5f8729-df67-4e02-b7c5-f2250efa9678\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2" Oct 30 13:15:00.541828 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.540571 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kcll\" (UniqueName: \"kubernetes.io/projected/0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b-kube-api-access-5kcll\") pod \"maas-api-key-cleanup-e2e-api-a-5b9e8e-29889435-6cs4n\" (UID: \"0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5b9e8e-29889435-6cs4n" Oct 30 13:15:00.541828 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.540601 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtj5s\" (UniqueName: \"kubernetes.io/projected/08e53b66-2198-4d37-82d5-563f36393d67-kube-api-access-rtj5s\") pod \"maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv\" (UID: \"08e53b66-2198-4d37-82d5-563f36393d67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv" Oct 30 13:15:00.541828 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.540701 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f06e3aa4-5df5-45c3-a80e-02b922c62768-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889435-h2hfn\" (UID: \"f06e3aa4-5df5-45c3-a80e-02b922c62768\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-h2hfn" Oct 30 13:15:00.541828 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.540731 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3bc7915-8ed0-47c5-a693-15db3d153fe8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj\" (UID: \"c3bc7915-8ed0-47c5-a693-15db3d153fe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj" Oct 30 13:15:00.541828 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.540766 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1814684-56ff-4b44-8328-29cbe0497bf2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-31e10b-29889435-tt6md\" (UID: \"f1814684-56ff-4b44-8328-29cbe0497bf2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31e10b-29889435-tt6md" Oct 30 13:15:00.541828 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.540868 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpdtt\" (UniqueName: \"kubernetes.io/projected/d882efb1-cd4f-45ce-b829-4795003e31b9-kube-api-access-cpdtt\") pod \"maas-api-key-cleanup-e2e-api-b-14e8a4-29889435-cldf2\" (UID: \"d882efb1-cd4f-45ce-b829-4795003e31b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-14e8a4-29889435-cldf2" Oct 30 13:15:00.541828 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.540912 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-5b9e8e-29889435-6cs4n\" (UID: \"0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5b9e8e-29889435-6cs4n" Oct 30 13:15:00.541828 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.540943 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/795f8222-5020-4ef8-be21-49fbb8b178e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889435-q4q4t\" (UID: \"795f8222-5020-4ef8-be21-49fbb8b178e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889435-q4q4t" Oct 30 13:15:00.541828 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.540980 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2e9458-537c-428a-abbe-6ff129af3c6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-4a8ee1-29889435-5w88z\" (UID: \"ec2e9458-537c-428a-abbe-6ff129af3c6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4a8ee1-29889435-5w88z" Oct 30 13:15:00.541828 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.541039 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffmq8\" (UniqueName: \"kubernetes.io/projected/f06e3aa4-5df5-45c3-a80e-02b922c62768-kube-api-access-ffmq8\") pod \"maas-api-key-cleanup-29889435-h2hfn\" (UID: \"f06e3aa4-5df5-45c3-a80e-02b922c62768\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-h2hfn" Oct 30 13:15:00.541828 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.541073 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw464\" (UniqueName: \"kubernetes.io/projected/ec2e9458-537c-428a-abbe-6ff129af3c6e-kube-api-access-xw464\") pod \"maas-api-key-cleanup-e2e-inf-a-4a8ee1-29889435-5w88z\" (UID: \"ec2e9458-537c-428a-abbe-6ff129af3c6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4a8ee1-29889435-5w88z" Oct 30 13:15:00.541828 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.541112 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a5f8729-df67-4e02-b7c5-f2250efa9678-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2\" (UID: \"2a5f8729-df67-4e02-b7c5-f2250efa9678\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2" Oct 30 13:15:00.541828 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.541145 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08e53b66-2198-4d37-82d5-563f36393d67-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv\" (UID: \"08e53b66-2198-4d37-82d5-563f36393d67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv" Oct 30 13:15:00.546197 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.544392 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3bc7915-8ed0-47c5-a693-15db3d153fe8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj\" (UID: \"c3bc7915-8ed0-47c5-a693-15db3d153fe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj" Oct 30 13:15:00.546197 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.545059 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/795f8222-5020-4ef8-be21-49fbb8b178e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889435-q4q4t\" (UID: \"795f8222-5020-4ef8-be21-49fbb8b178e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889435-q4q4t" Oct 30 13:15:00.546197 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.546144 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a5f8729-df67-4e02-b7c5-f2250efa9678-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2\" (UID: \"2a5f8729-df67-4e02-b7c5-f2250efa9678\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2" Oct 30 13:15:00.560560 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.560528 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh" Oct 30 13:15:00.571302 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.570697 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889435-h2hfn"] Oct 30 13:15:00.587400 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.587364 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-14e8a4-29889435-cldf2"] Oct 30 13:15:00.600449 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.600378 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktxdj\" (UniqueName: \"kubernetes.io/projected/795f8222-5020-4ef8-be21-49fbb8b178e8-kube-api-access-ktxdj\") pod \"maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889435-q4q4t\" (UID: \"795f8222-5020-4ef8-be21-49fbb8b178e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889435-q4q4t" Oct 30 13:15:00.608177 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.608121 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpxxf\" (UniqueName: \"kubernetes.io/projected/c3bc7915-8ed0-47c5-a693-15db3d153fe8-kube-api-access-tpxxf\") pod \"maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj\" (UID: \"c3bc7915-8ed0-47c5-a693-15db3d153fe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj" Oct 30 13:15:00.608871 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.608729 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ggzk\" (UniqueName: \"kubernetes.io/projected/2a5f8729-df67-4e02-b7c5-f2250efa9678-kube-api-access-2ggzk\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2\" (UID: \"2a5f8729-df67-4e02-b7c5-f2250efa9678\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2" Oct 30 13:15:00.613451 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.613425 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2" Oct 30 13:15:00.650552 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.645179 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1814684-56ff-4b44-8328-29cbe0497bf2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-31e10b-29889435-tt6md\" (UID: \"f1814684-56ff-4b44-8328-29cbe0497bf2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31e10b-29889435-tt6md" Oct 30 13:15:00.650552 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.645300 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpdtt\" (UniqueName: \"kubernetes.io/projected/d882efb1-cd4f-45ce-b829-4795003e31b9-kube-api-access-cpdtt\") pod \"maas-api-key-cleanup-e2e-api-b-14e8a4-29889435-cldf2\" (UID: \"d882efb1-cd4f-45ce-b829-4795003e31b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-14e8a4-29889435-cldf2" Oct 30 13:15:00.650552 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.645346 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-5b9e8e-29889435-6cs4n\" (UID: \"0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5b9e8e-29889435-6cs4n" Oct 30 13:15:00.651326 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.650890 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2e9458-537c-428a-abbe-6ff129af3c6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-4a8ee1-29889435-5w88z\" (UID: \"ec2e9458-537c-428a-abbe-6ff129af3c6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4a8ee1-29889435-5w88z" Oct 30 13:15:00.651326 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.650961 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffmq8\" (UniqueName: \"kubernetes.io/projected/f06e3aa4-5df5-45c3-a80e-02b922c62768-kube-api-access-ffmq8\") pod \"maas-api-key-cleanup-29889435-h2hfn\" (UID: \"f06e3aa4-5df5-45c3-a80e-02b922c62768\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-h2hfn" Oct 30 13:15:00.651326 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.650994 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw464\" (UniqueName: \"kubernetes.io/projected/ec2e9458-537c-428a-abbe-6ff129af3c6e-kube-api-access-xw464\") pod \"maas-api-key-cleanup-e2e-inf-a-4a8ee1-29889435-5w88z\" (UID: \"ec2e9458-537c-428a-abbe-6ff129af3c6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4a8ee1-29889435-5w88z" Oct 30 13:15:00.651326 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.651041 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08e53b66-2198-4d37-82d5-563f36393d67-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv\" (UID: \"08e53b66-2198-4d37-82d5-563f36393d67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv" Oct 30 13:15:00.651326 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.651112 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khkhv\" (UniqueName: \"kubernetes.io/projected/f1814684-56ff-4b44-8328-29cbe0497bf2-kube-api-access-khkhv\") pod \"maas-api-key-cleanup-e2e-api-b-31e10b-29889435-tt6md\" (UID: \"f1814684-56ff-4b44-8328-29cbe0497bf2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31e10b-29889435-tt6md" Oct 30 13:15:00.651326 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.651195 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d882efb1-cd4f-45ce-b829-4795003e31b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-14e8a4-29889435-cldf2\" (UID: \"d882efb1-cd4f-45ce-b829-4795003e31b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-14e8a4-29889435-cldf2" Oct 30 13:15:00.652436 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.651799 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kcll\" (UniqueName: \"kubernetes.io/projected/0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b-kube-api-access-5kcll\") pod \"maas-api-key-cleanup-e2e-api-a-5b9e8e-29889435-6cs4n\" (UID: \"0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5b9e8e-29889435-6cs4n" Oct 30 13:15:00.652436 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.651858 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtj5s\" (UniqueName: \"kubernetes.io/projected/08e53b66-2198-4d37-82d5-563f36393d67-kube-api-access-rtj5s\") pod \"maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv\" (UID: \"08e53b66-2198-4d37-82d5-563f36393d67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv" Oct 30 13:15:00.652436 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.651968 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f06e3aa4-5df5-45c3-a80e-02b922c62768-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889435-h2hfn\" (UID: \"f06e3aa4-5df5-45c3-a80e-02b922c62768\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-h2hfn" Oct 30 13:15:00.659338 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.653363 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1814684-56ff-4b44-8328-29cbe0497bf2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-31e10b-29889435-tt6md\" (UID: \"f1814684-56ff-4b44-8328-29cbe0497bf2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31e10b-29889435-tt6md" Oct 30 13:15:00.659338 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.657384 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08e53b66-2198-4d37-82d5-563f36393d67-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv\" (UID: \"08e53b66-2198-4d37-82d5-563f36393d67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv" Oct 30 13:15:00.659338 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.658145 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-5b9e8e-29889435-6cs4n\" (UID: \"0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5b9e8e-29889435-6cs4n" Oct 30 13:15:00.659338 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.658633 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2e9458-537c-428a-abbe-6ff129af3c6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-4a8ee1-29889435-5w88z\" (UID: \"ec2e9458-537c-428a-abbe-6ff129af3c6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4a8ee1-29889435-5w88z" Oct 30 13:15:00.660001 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.659953 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d882efb1-cd4f-45ce-b829-4795003e31b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-14e8a4-29889435-cldf2\" (UID: \"d882efb1-cd4f-45ce-b829-4795003e31b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-14e8a4-29889435-cldf2" Oct 30 13:15:00.660900 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.660632 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f06e3aa4-5df5-45c3-a80e-02b922c62768-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889435-h2hfn\" (UID: \"f06e3aa4-5df5-45c3-a80e-02b922c62768\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-h2hfn" Oct 30 13:15:00.707966 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.707878 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpdtt\" (UniqueName: \"kubernetes.io/projected/d882efb1-cd4f-45ce-b829-4795003e31b9-kube-api-access-cpdtt\") pod \"maas-api-key-cleanup-e2e-api-b-14e8a4-29889435-cldf2\" (UID: \"d882efb1-cd4f-45ce-b829-4795003e31b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-14e8a4-29889435-cldf2" Oct 30 13:15:00.723320 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.722008 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khkhv\" (UniqueName: \"kubernetes.io/projected/f1814684-56ff-4b44-8328-29cbe0497bf2-kube-api-access-khkhv\") pod \"maas-api-key-cleanup-e2e-api-b-31e10b-29889435-tt6md\" (UID: \"f1814684-56ff-4b44-8328-29cbe0497bf2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31e10b-29889435-tt6md" Oct 30 13:15:00.723320 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.722821 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtj5s\" (UniqueName: \"kubernetes.io/projected/08e53b66-2198-4d37-82d5-563f36393d67-kube-api-access-rtj5s\") pod \"maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv\" (UID: \"08e53b66-2198-4d37-82d5-563f36393d67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv" Oct 30 13:15:00.727329 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.725297 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw464\" (UniqueName: \"kubernetes.io/projected/ec2e9458-537c-428a-abbe-6ff129af3c6e-kube-api-access-xw464\") pod \"maas-api-key-cleanup-e2e-inf-a-4a8ee1-29889435-5w88z\" (UID: \"ec2e9458-537c-428a-abbe-6ff129af3c6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4a8ee1-29889435-5w88z" Oct 30 13:15:00.727329 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.725661 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffmq8\" (UniqueName: \"kubernetes.io/projected/f06e3aa4-5df5-45c3-a80e-02b922c62768-kube-api-access-ffmq8\") pod \"maas-api-key-cleanup-29889435-h2hfn\" (UID: \"f06e3aa4-5df5-45c3-a80e-02b922c62768\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-h2hfn" Oct 30 13:15:00.743161 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.743129 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kcll\" (UniqueName: \"kubernetes.io/projected/0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b-kube-api-access-5kcll\") pod \"maas-api-key-cleanup-e2e-api-a-5b9e8e-29889435-6cs4n\" (UID: \"0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5b9e8e-29889435-6cs4n" Oct 30 13:15:00.760177 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.759915 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4a8ee1-29889435-5w88z" Oct 30 13:15:00.775637 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.775476 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv" Oct 30 13:15:00.783641 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.783600 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31e10b-29889435-tt6md" Oct 30 13:15:00.890675 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.890445 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889435-q4q4t" Oct 30 13:15:00.896568 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.896480 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-h2hfn" Oct 30 13:15:00.905020 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.904183 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj" Oct 30 13:15:00.905020 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.904236 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj"] Oct 30 13:15:00.961582 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.961555 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5b9e8e-29889435-6cs4n" Oct 30 13:15:00.962994 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.962959 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-14e8a4-29889435-cldf2" Oct 30 13:15:00.999772 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:00.999737 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh"] Oct 30 13:15:01.031946 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:01.031915 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2"] Oct 30 13:15:01.139818 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:01.139788 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4a8ee1-29889435-5w88z"] Oct 30 13:15:01.149549 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:15:01.149323 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2e9458_537c_428a_abbe_6ff129af3c6e.slice/crio-f7fa618e7d6e1b6ceff3a0f8301032fea00d82d34ae6846c88d0f7486bca31d3 WatchSource:0}: Error finding container f7fa618e7d6e1b6ceff3a0f8301032fea00d82d34ae6846c88d0f7486bca31d3: Status 404 returned error can't find the container with id f7fa618e7d6e1b6ceff3a0f8301032fea00d82d34ae6846c88d0f7486bca31d3 Oct 30 13:15:01.206044 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:01.205947 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31e10b-29889435-tt6md"] Oct 30 13:15:01.213633 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:01.213605 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv"] Oct 30 13:15:01.246960 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:15:01.246698 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08e53b66_2198_4d37_82d5_563f36393d67.slice/crio-5f3b834b341aadef5ca28b0322aea7db2f62c71268fcbd54e423d6884b4629d4 WatchSource:0}: Error finding container 5f3b834b341aadef5ca28b0322aea7db2f62c71268fcbd54e423d6884b4629d4: Status 404 returned error can't find the container with id 5f3b834b341aadef5ca28b0322aea7db2f62c71268fcbd54e423d6884b4629d4 Oct 30 13:15:01.683487 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:01.682626 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889435-h2hfn"] Oct 30 13:15:01.692779 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:01.692553 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889435-q4q4t"] Oct 30 13:15:01.697624 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:01.697547 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh" event={"ID":"01a78c34-1e58-48f3-8660-56d6b19d8847","Type":"ContainerStarted","Data":"20b3cb4786b13a323e7db741aaeedff259324c672c25c1b9092e03a677f84e3f"} Oct 30 13:15:01.702529 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:15:01.702354 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf06e3aa4_5df5_45c3_a80e_02b922c62768.slice/crio-ec0d9de09e50436fb27fdca1be194030530b372864a36e1d419bb24fbcb2c62f WatchSource:0}: Error finding container ec0d9de09e50436fb27fdca1be194030530b372864a36e1d419bb24fbcb2c62f: Status 404 returned error can't find the container with id ec0d9de09e50436fb27fdca1be194030530b372864a36e1d419bb24fbcb2c62f Oct 30 13:15:01.703062 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:01.703023 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj" event={"ID":"ea3c064a-c226-4eec-bd7e-d42fb03965ad","Type":"ContainerStarted","Data":"d5983de0b3f97ee8a05e3a431d8ef8ef5189200ad88d94109d4fa78607378fd4"} Oct 30 13:15:01.705109 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:15:01.705085 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795f8222_5020_4ef8_be21_49fbb8b178e8.slice/crio-2dd3c2cd8bebce9ae76a506d53730e16a15b7476a960f8d0006db2c3e5c2296c WatchSource:0}: Error finding container 2dd3c2cd8bebce9ae76a506d53730e16a15b7476a960f8d0006db2c3e5c2296c: Status 404 returned error can't find the container with id 2dd3c2cd8bebce9ae76a506d53730e16a15b7476a960f8d0006db2c3e5c2296c Oct 30 13:15:01.708863 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:01.708832 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31e10b-29889435-tt6md" event={"ID":"f1814684-56ff-4b44-8328-29cbe0497bf2","Type":"ContainerStarted","Data":"33735a3e149e4a6069dd9ed1df627b815ddcda02475bb43e8cbeb1db50c59800"} Oct 30 13:15:01.715302 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:01.714819 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4a8ee1-29889435-5w88z" event={"ID":"ec2e9458-537c-428a-abbe-6ff129af3c6e","Type":"ContainerStarted","Data":"f7fa618e7d6e1b6ceff3a0f8301032fea00d82d34ae6846c88d0f7486bca31d3"} Oct 30 13:15:01.717397 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:01.717354 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv" event={"ID":"08e53b66-2198-4d37-82d5-563f36393d67","Type":"ContainerStarted","Data":"5f3b834b341aadef5ca28b0322aea7db2f62c71268fcbd54e423d6884b4629d4"} Oct 30 13:15:01.719448 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:01.719426 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2" event={"ID":"2a5f8729-df67-4e02-b7c5-f2250efa9678","Type":"ContainerStarted","Data":"23900db97ceb9dc12b5c70cb68c4508dc57fd163d5aae5a50a089d8025494f7d"} Oct 30 13:15:01.788330 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:01.788254 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2" podStartSLOduration=1.7882349560000002 podStartE2EDuration="1.788234956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:01.78482675 +0000 UTC m=+1800.388325197" watchObservedRunningTime="2026-10-30 13:15:01.788234956 +0000 UTC m=+1800.391733402" Oct 30 13:15:01.843578 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:01.843544 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-14e8a4-29889435-cldf2"] Oct 30 13:15:01.873718 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:01.873687 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/2.log" Oct 30 13:15:01.881149 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:01.881114 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/2.log" Oct 30 13:15:01.941730 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:15:01.941693 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd882efb1_cd4f_45ce_b829_4795003e31b9.slice/crio-43bb5c6868b99cbc1e7305441d4e03d39028879a995f7094f638aa715164a756 WatchSource:0}: Error finding container 43bb5c6868b99cbc1e7305441d4e03d39028879a995f7094f638aa715164a756: Status 404 returned error can't find the container with id 43bb5c6868b99cbc1e7305441d4e03d39028879a995f7094f638aa715164a756 Oct 30 13:15:01.945622 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:01.945564 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj"] Oct 30 13:15:01.950404 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:01.950378 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5b9e8e-29889435-6cs4n"] Oct 30 13:15:02.277589 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:02.277551 4931 scope.go:122] "RemoveContainer" containerID="550ee4479e76ad0efb2952086dd83b1277d42eb66ab1adc0c5d524053170dfb1" Oct 30 13:15:02.728784 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:02.728752 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-h2hfn" event={"ID":"f06e3aa4-5df5-45c3-a80e-02b922c62768","Type":"ContainerStarted","Data":"879cdb4f8d7fff02d514afeae8a196413e02e877048ba98695504b27a3fb902d"} Oct 30 13:15:02.728957 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:02.728804 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-h2hfn" event={"ID":"f06e3aa4-5df5-45c3-a80e-02b922c62768","Type":"ContainerStarted","Data":"ec0d9de09e50436fb27fdca1be194030530b372864a36e1d419bb24fbcb2c62f"} Oct 30 13:15:02.730486 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:02.730463 4931 generic.go:356] "Generic (PLEG): container finished" podID="2a5f8729-df67-4e02-b7c5-f2250efa9678" containerID="621a4987c033588eb280722c252550412e34b7b15dbb6dc472fde1475501ba90" exitCode=60 Oct 30 13:15:02.730582 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:02.730552 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2" event={"ID":"2a5f8729-df67-4e02-b7c5-f2250efa9678","Type":"ContainerDied","Data":"621a4987c033588eb280722c252550412e34b7b15dbb6dc472fde1475501ba90"} Oct 30 13:15:02.730759 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:02.730743 4931 scope.go:122] "RemoveContainer" containerID="621a4987c033588eb280722c252550412e34b7b15dbb6dc472fde1475501ba90" Oct 30 13:15:02.732144 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:02.732120 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj" event={"ID":"c3bc7915-8ed0-47c5-a693-15db3d153fe8","Type":"ContainerStarted","Data":"da5fd92b60a324539bd8653988db5e29ce4e017bae6356ececad22b32ee02599"} Oct 30 13:15:02.732241 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:02.732162 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj" event={"ID":"c3bc7915-8ed0-47c5-a693-15db3d153fe8","Type":"ContainerStarted","Data":"e21c3079756b17c59380f55db9f63ae31853bbcdedaa5ccffcc3b79bfa9a948b"} Oct 30 13:15:02.737305 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:02.737259 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5b9e8e-29889435-6cs4n" event={"ID":"0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b","Type":"ContainerStarted","Data":"e1803b357afb6d8d2121a62e2eb228dff1b8e61eb86b01802b257f6962af1b67"} Oct 30 13:15:02.737425 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:02.737324 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5b9e8e-29889435-6cs4n" event={"ID":"0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b","Type":"ContainerStarted","Data":"b0cbadd3a22a8cdd6f060a7e6ee178260ba00be51f0a95491fbc5a50b44a264a"} Oct 30 13:15:02.739654 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:02.739633 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-14e8a4-29889435-cldf2" event={"ID":"d882efb1-cd4f-45ce-b829-4795003e31b9","Type":"ContainerStarted","Data":"15898ffe8fa496be25d803a19667bec970fe86893a715188569c0a2f074e5b65"} Oct 30 13:15:02.739748 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:02.739668 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-14e8a4-29889435-cldf2" event={"ID":"d882efb1-cd4f-45ce-b829-4795003e31b9","Type":"ContainerStarted","Data":"43bb5c6868b99cbc1e7305441d4e03d39028879a995f7094f638aa715164a756"} Oct 30 13:15:02.742097 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:02.742074 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4a8ee1-29889435-5w88z" event={"ID":"ec2e9458-537c-428a-abbe-6ff129af3c6e","Type":"ContainerStarted","Data":"2f972d748ef051be6bdb94b18c2ecccf7496e0c0b1fe5f8ff44fdd2b88a61f21"} Oct 30 13:15:02.743756 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:02.743731 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889435-q4q4t" event={"ID":"795f8222-5020-4ef8-be21-49fbb8b178e8","Type":"ContainerStarted","Data":"abd5554a8570829d92db46b64902475716a3ca59311249a14f5089f3cc24d53d"} Oct 30 13:15:02.743835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:02.743777 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889435-q4q4t" event={"ID":"795f8222-5020-4ef8-be21-49fbb8b178e8","Type":"ContainerStarted","Data":"2dd3c2cd8bebce9ae76a506d53730e16a15b7476a960f8d0006db2c3e5c2296c"} Oct 30 13:15:02.745254 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:02.745235 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv" event={"ID":"08e53b66-2198-4d37-82d5-563f36393d67","Type":"ContainerStarted","Data":"e7059497e0281af8fd7ae3bbb128532f2477a49509bdad1dce072cc652a9287a"} Oct 30 13:15:02.746853 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:02.746834 4931 generic.go:356] "Generic (PLEG): container finished" podID="01a78c34-1e58-48f3-8660-56d6b19d8847" containerID="aea2186ee0e842503083f5b8fa07351a332f9c8f983598e378c4449e8b32fa9d" exitCode=60 Oct 30 13:15:02.746937 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:02.746904 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh" event={"ID":"01a78c34-1e58-48f3-8660-56d6b19d8847","Type":"ContainerDied","Data":"aea2186ee0e842503083f5b8fa07351a332f9c8f983598e378c4449e8b32fa9d"} Oct 30 13:15:02.747089 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:02.747074 4931 scope.go:122] "RemoveContainer" containerID="aea2186ee0e842503083f5b8fa07351a332f9c8f983598e378c4449e8b32fa9d" Oct 30 13:15:02.748624 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:02.748531 4931 generic.go:356] "Generic (PLEG): container finished" podID="ea3c064a-c226-4eec-bd7e-d42fb03965ad" containerID="6111a7bd0ee640a6f6a0d2e7ec7bc02969bd041be711f15e7f3d68789eb9ca1c" exitCode=60 Oct 30 13:15:02.748624 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:02.748612 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj" event={"ID":"ea3c064a-c226-4eec-bd7e-d42fb03965ad","Type":"ContainerDied","Data":"6111a7bd0ee640a6f6a0d2e7ec7bc02969bd041be711f15e7f3d68789eb9ca1c"} Oct 30 13:15:02.748811 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:02.748792 4931 scope.go:122] "RemoveContainer" containerID="6111a7bd0ee640a6f6a0d2e7ec7bc02969bd041be711f15e7f3d68789eb9ca1c" Oct 30 13:15:02.750811 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:02.750790 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31e10b-29889435-tt6md" event={"ID":"f1814684-56ff-4b44-8328-29cbe0497bf2","Type":"ContainerStarted","Data":"e79b17db4eef990f94670cd9efe3642c428c2d985a304493cca2d814d4a3c3f0"} Oct 30 13:15:02.776359 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:02.776316 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-h2hfn" podStartSLOduration=2.776304356 podStartE2EDuration="2.776304356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:02.775127478 +0000 UTC m=+1801.378625923" watchObservedRunningTime="2026-10-30 13:15:02.776304356 +0000 UTC m=+1801.379802798" Oct 30 13:15:02.829815 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:02.829772 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv" podStartSLOduration=2.829759171 podStartE2EDuration="2.829759171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:02.824960168 +0000 UTC m=+1801.428458624" watchObservedRunningTime="2026-10-30 13:15:02.829759171 +0000 UTC m=+1801.433257616" Oct 30 13:15:02.917750 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:02.917695 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889435-q4q4t" podStartSLOduration=2.9176813150000003 podStartE2EDuration="2.917681315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:02.915694184 +0000 UTC m=+1801.519192630" watchObservedRunningTime="2026-10-30 13:15:02.917681315 +0000 UTC m=+1801.521179761" Oct 30 13:15:03.003598 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:03.003545 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-14e8a4-29889435-cldf2" podStartSLOduration=3.00353185 podStartE2EDuration="3.00353185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:03.00099343 +0000 UTC m=+1801.604491906" watchObservedRunningTime="2026-10-30 13:15:03.00353185 +0000 UTC m=+1801.607030295" Oct 30 13:15:03.090380 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:03.090131 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31e10b-29889435-tt6md" podStartSLOduration=3.090106479 podStartE2EDuration="3.090106479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:03.054707459 +0000 UTC m=+1801.658205925" watchObservedRunningTime="2026-10-30 13:15:03.090106479 +0000 UTC m=+1801.693604926" Oct 30 13:15:03.176328 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:03.173126 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:15:03.186386 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:03.186043 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5b9e8e-29889435-6cs4n" podStartSLOduration=3.186021523 podStartE2EDuration="3.186021523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:03.154202115 +0000 UTC m=+1801.757700562" watchObservedRunningTime="2026-10-30 13:15:03.186021523 +0000 UTC m=+1801.789519971" Oct 30 13:15:03.196814 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:03.196746 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj" podStartSLOduration=3.19672777 podStartE2EDuration="3.19672777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:03.195883776 +0000 UTC m=+1801.799382252" watchObservedRunningTime="2026-10-30 13:15:03.19672777 +0000 UTC m=+1801.800226217" Oct 30 13:15:03.242411 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:03.242355 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4a8ee1-29889435-5w88z" podStartSLOduration=3.242336097 podStartE2EDuration="3.242336097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:03.241709347 +0000 UTC m=+1801.845207795" watchObservedRunningTime="2026-10-30 13:15:03.242336097 +0000 UTC m=+1801.845834542" Oct 30 13:15:03.330602 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:15:03.325257 4931 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd882efb1_cd4f_45ce_b829_4795003e31b9.slice/crio-conmon-15898ffe8fa496be25d803a19667bec970fe86893a715188569c0a2f074e5b65.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd882efb1_cd4f_45ce_b829_4795003e31b9.slice/crio-15898ffe8fa496be25d803a19667bec970fe86893a715188569c0a2f074e5b65.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3bc7915_8ed0_47c5_a693_15db3d153fe8.slice/crio-da5fd92b60a324539bd8653988db5e29ce4e017bae6356ececad22b32ee02599.scope\": RecentStats: unable to find data in memory cache]" Oct 30 13:15:03.759718 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:03.759687 4931 generic.go:356] "Generic (PLEG): container finished" podID="c3bc7915-8ed0-47c5-a693-15db3d153fe8" containerID="da5fd92b60a324539bd8653988db5e29ce4e017bae6356ececad22b32ee02599" exitCode=60 Oct 30 13:15:03.759891 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:03.759777 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj" event={"ID":"c3bc7915-8ed0-47c5-a693-15db3d153fe8","Type":"ContainerDied","Data":"da5fd92b60a324539bd8653988db5e29ce4e017bae6356ececad22b32ee02599"} Oct 30 13:15:03.760100 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:03.760082 4931 scope.go:122] "RemoveContainer" containerID="da5fd92b60a324539bd8653988db5e29ce4e017bae6356ececad22b32ee02599" Oct 30 13:15:03.761549 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:03.761532 4931 generic.go:356] "Generic (PLEG): container finished" podID="0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b" containerID="e1803b357afb6d8d2121a62e2eb228dff1b8e61eb86b01802b257f6962af1b67" exitCode=60 Oct 30 13:15:03.761638 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:03.761616 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5b9e8e-29889435-6cs4n" event={"ID":"0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b","Type":"ContainerDied","Data":"e1803b357afb6d8d2121a62e2eb228dff1b8e61eb86b01802b257f6962af1b67"} Oct 30 13:15:03.761975 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:03.761907 4931 scope.go:122] "RemoveContainer" containerID="e1803b357afb6d8d2121a62e2eb228dff1b8e61eb86b01802b257f6962af1b67" Oct 30 13:15:03.763607 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:03.763583 4931 generic.go:356] "Generic (PLEG): container finished" podID="d882efb1-cd4f-45ce-b829-4795003e31b9" containerID="15898ffe8fa496be25d803a19667bec970fe86893a715188569c0a2f074e5b65" exitCode=60 Oct 30 13:15:03.763698 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:03.763658 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-14e8a4-29889435-cldf2" event={"ID":"d882efb1-cd4f-45ce-b829-4795003e31b9","Type":"ContainerDied","Data":"15898ffe8fa496be25d803a19667bec970fe86893a715188569c0a2f074e5b65"} Oct 30 13:15:03.763896 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:03.763876 4931 scope.go:122] "RemoveContainer" containerID="15898ffe8fa496be25d803a19667bec970fe86893a715188569c0a2f074e5b65" Oct 30 13:15:03.765997 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:03.765972 4931 generic.go:356] "Generic (PLEG): container finished" podID="ec2e9458-537c-428a-abbe-6ff129af3c6e" containerID="2f972d748ef051be6bdb94b18c2ecccf7496e0c0b1fe5f8ff44fdd2b88a61f21" exitCode=60 Oct 30 13:15:03.766141 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:03.766094 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4a8ee1-29889435-5w88z" event={"ID":"ec2e9458-537c-428a-abbe-6ff129af3c6e","Type":"ContainerDied","Data":"2f972d748ef051be6bdb94b18c2ecccf7496e0c0b1fe5f8ff44fdd2b88a61f21"} Oct 30 13:15:03.766528 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:03.766509 4931 scope.go:122] "RemoveContainer" containerID="2f972d748ef051be6bdb94b18c2ecccf7496e0c0b1fe5f8ff44fdd2b88a61f21" Oct 30 13:15:03.767947 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:03.767929 4931 generic.go:356] "Generic (PLEG): container finished" podID="795f8222-5020-4ef8-be21-49fbb8b178e8" containerID="abd5554a8570829d92db46b64902475716a3ca59311249a14f5089f3cc24d53d" exitCode=60 Oct 30 13:15:03.768030 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:03.768013 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889435-q4q4t" event={"ID":"795f8222-5020-4ef8-be21-49fbb8b178e8","Type":"ContainerDied","Data":"abd5554a8570829d92db46b64902475716a3ca59311249a14f5089f3cc24d53d"} Oct 30 13:15:03.768195 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:03.768179 4931 scope.go:122] "RemoveContainer" containerID="abd5554a8570829d92db46b64902475716a3ca59311249a14f5089f3cc24d53d" Oct 30 13:15:03.769859 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:03.769840 4931 generic.go:356] "Generic (PLEG): container finished" podID="08e53b66-2198-4d37-82d5-563f36393d67" containerID="e7059497e0281af8fd7ae3bbb128532f2477a49509bdad1dce072cc652a9287a" exitCode=60 Oct 30 13:15:03.769954 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:03.769918 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv" event={"ID":"08e53b66-2198-4d37-82d5-563f36393d67","Type":"ContainerDied","Data":"e7059497e0281af8fd7ae3bbb128532f2477a49509bdad1dce072cc652a9287a"} Oct 30 13:15:03.770117 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:03.770103 4931 scope.go:122] "RemoveContainer" containerID="e7059497e0281af8fd7ae3bbb128532f2477a49509bdad1dce072cc652a9287a" Oct 30 13:15:03.771928 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:03.771907 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh" event={"ID":"01a78c34-1e58-48f3-8660-56d6b19d8847","Type":"ContainerStarted","Data":"d3bf2e6f3d52109bfbad1217c9cbe9b9d5c4dd6adc5963beaa7aba296969dc55"} Oct 30 13:15:03.774806 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:03.774787 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj" event={"ID":"ea3c064a-c226-4eec-bd7e-d42fb03965ad","Type":"ContainerStarted","Data":"4cb9280b9412719c415c6732759e3cfe1f6472e49a992928646cf269616ffc76"} Oct 30 13:15:03.776583 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:03.776563 4931 generic.go:356] "Generic (PLEG): container finished" podID="f1814684-56ff-4b44-8328-29cbe0497bf2" containerID="e79b17db4eef990f94670cd9efe3642c428c2d985a304493cca2d814d4a3c3f0" exitCode=60 Oct 30 13:15:03.776678 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:03.776593 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31e10b-29889435-tt6md" event={"ID":"f1814684-56ff-4b44-8328-29cbe0497bf2","Type":"ContainerDied","Data":"e79b17db4eef990f94670cd9efe3642c428c2d985a304493cca2d814d4a3c3f0"} Oct 30 13:15:03.776855 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:03.776837 4931 scope.go:122] "RemoveContainer" containerID="e79b17db4eef990f94670cd9efe3642c428c2d985a304493cca2d814d4a3c3f0" Oct 30 13:15:03.778956 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:03.778933 4931 generic.go:356] "Generic (PLEG): container finished" podID="f06e3aa4-5df5-45c3-a80e-02b922c62768" containerID="879cdb4f8d7fff02d514afeae8a196413e02e877048ba98695504b27a3fb902d" exitCode=60 Oct 30 13:15:03.779031 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:03.778980 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-h2hfn" event={"ID":"f06e3aa4-5df5-45c3-a80e-02b922c62768","Type":"ContainerDied","Data":"879cdb4f8d7fff02d514afeae8a196413e02e877048ba98695504b27a3fb902d"} Oct 30 13:15:03.779242 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:03.779224 4931 scope.go:122] "RemoveContainer" containerID="879cdb4f8d7fff02d514afeae8a196413e02e877048ba98695504b27a3fb902d" Oct 30 13:15:03.781139 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:03.781119 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2" event={"ID":"2a5f8729-df67-4e02-b7c5-f2250efa9678","Type":"ContainerStarted","Data":"064d14caeb9e6b558442ac8ca3b1971bd1d69c54808471b82fbf38f7f6e322f9"} Oct 30 13:15:04.017173 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:04.017074 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh" podStartSLOduration=4.017060042 podStartE2EDuration="4.017060042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:04.014524868 +0000 UTC m=+1802.618023329" watchObservedRunningTime="2026-10-30 13:15:04.017060042 +0000 UTC m=+1802.620558487" Oct 30 13:15:04.216930 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:04.203525 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj" podStartSLOduration=4.203504879 podStartE2EDuration="4.203504879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:04.192869999 +0000 UTC m=+1802.796368453" watchObservedRunningTime="2026-10-30 13:15:04.203504879 +0000 UTC m=+1802.807003325" Oct 30 13:15:04.832133 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:04.827374 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv" event={"ID":"08e53b66-2198-4d37-82d5-563f36393d67","Type":"ContainerStarted","Data":"5388bbb1ed5edc19ba11d5ad380d23d471f2dbbce0fdea098f29d4e36a2b9990"} Oct 30 13:15:04.859044 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:04.857861 4931 generic.go:356] "Generic (PLEG): container finished" podID="01a78c34-1e58-48f3-8660-56d6b19d8847" containerID="d3bf2e6f3d52109bfbad1217c9cbe9b9d5c4dd6adc5963beaa7aba296969dc55" exitCode=60 Oct 30 13:15:04.859044 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:04.858419 4931 scope.go:122] "RemoveContainer" containerID="d3bf2e6f3d52109bfbad1217c9cbe9b9d5c4dd6adc5963beaa7aba296969dc55" Oct 30 13:15:04.859044 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:15:04.858812 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh_odh-ai-gateway-infra(01a78c34-1e58-48f3-8660-56d6b19d8847)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh" podUID="01a78c34-1e58-48f3-8660-56d6b19d8847" Oct 30 13:15:04.859044 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:04.858861 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh" event={"ID":"01a78c34-1e58-48f3-8660-56d6b19d8847","Type":"ContainerDied","Data":"d3bf2e6f3d52109bfbad1217c9cbe9b9d5c4dd6adc5963beaa7aba296969dc55"} Oct 30 13:15:04.859044 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:04.858912 4931 scope.go:122] "RemoveContainer" containerID="aea2186ee0e842503083f5b8fa07351a332f9c8f983598e378c4449e8b32fa9d" Oct 30 13:15:04.873639 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:04.872853 4931 generic.go:356] "Generic (PLEG): container finished" podID="ea3c064a-c226-4eec-bd7e-d42fb03965ad" containerID="4cb9280b9412719c415c6732759e3cfe1f6472e49a992928646cf269616ffc76" exitCode=60 Oct 30 13:15:04.873639 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:04.872953 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj" event={"ID":"ea3c064a-c226-4eec-bd7e-d42fb03965ad","Type":"ContainerDied","Data":"4cb9280b9412719c415c6732759e3cfe1f6472e49a992928646cf269616ffc76"} Oct 30 13:15:04.873639 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:04.873398 4931 scope.go:122] "RemoveContainer" containerID="4cb9280b9412719c415c6732759e3cfe1f6472e49a992928646cf269616ffc76" Oct 30 13:15:04.882245 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:15:04.874811 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj_odh-ai-gateway-infra(ea3c064a-c226-4eec-bd7e-d42fb03965ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj" podUID="ea3c064a-c226-4eec-bd7e-d42fb03965ad" Oct 30 13:15:04.888505 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:04.888463 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-h2hfn" event={"ID":"f06e3aa4-5df5-45c3-a80e-02b922c62768","Type":"ContainerStarted","Data":"34ba8f6af56ac562bbd3dae00f48a6cafdfe60fadb1b551523729c93d136108f"} Oct 30 13:15:04.893459 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:04.892720 4931 generic.go:356] "Generic (PLEG): container finished" podID="2a5f8729-df67-4e02-b7c5-f2250efa9678" containerID="064d14caeb9e6b558442ac8ca3b1971bd1d69c54808471b82fbf38f7f6e322f9" exitCode=60 Oct 30 13:15:04.893459 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:04.892823 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2" event={"ID":"2a5f8729-df67-4e02-b7c5-f2250efa9678","Type":"ContainerDied","Data":"064d14caeb9e6b558442ac8ca3b1971bd1d69c54808471b82fbf38f7f6e322f9"} Oct 30 13:15:04.893459 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:04.893190 4931 scope.go:122] "RemoveContainer" containerID="064d14caeb9e6b558442ac8ca3b1971bd1d69c54808471b82fbf38f7f6e322f9" Oct 30 13:15:04.893915 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:15:04.893849 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2_odh-ai-gateway-infra(2a5f8729-df67-4e02-b7c5-f2250efa9678)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2" podUID="2a5f8729-df67-4e02-b7c5-f2250efa9678" Oct 30 13:15:04.896215 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:04.896190 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj" event={"ID":"c3bc7915-8ed0-47c5-a693-15db3d153fe8","Type":"ContainerStarted","Data":"8bd26f344bfab5e67d4561e09103026fec443584da08d4db313d91d1baf6f471"} Oct 30 13:15:04.919489 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:04.916715 4931 scope.go:122] "RemoveContainer" containerID="6111a7bd0ee640a6f6a0d2e7ec7bc02969bd041be711f15e7f3d68789eb9ca1c" Oct 30 13:15:04.919489 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:04.918370 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889435-q4q4t" event={"ID":"795f8222-5020-4ef8-be21-49fbb8b178e8","Type":"ContainerStarted","Data":"81a563c30bab377fb31f64628df2799df51e6729a7db27da9700da0525c1f6ea"} Oct 30 13:15:04.942064 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:04.942040 4931 scope.go:122] "RemoveContainer" containerID="621a4987c033588eb280722c252550412e34b7b15dbb6dc472fde1475501ba90" Oct 30 13:15:05.944907 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:05.944869 4931 scope.go:122] "RemoveContainer" containerID="4cb9280b9412719c415c6732759e3cfe1f6472e49a992928646cf269616ffc76" Oct 30 13:15:05.945411 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:15:05.945334 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj_odh-ai-gateway-infra(ea3c064a-c226-4eec-bd7e-d42fb03965ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj" podUID="ea3c064a-c226-4eec-bd7e-d42fb03965ad" Oct 30 13:15:05.947180 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:05.947144 4931 generic.go:356] "Generic (PLEG): container finished" podID="f1814684-56ff-4b44-8328-29cbe0497bf2" containerID="c21741ff7f845959397e5d29d924dd1521c375ae3477381719c81a89f4471717" exitCode=60 Oct 30 13:15:05.947361 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:05.947341 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31e10b-29889435-tt6md" event={"ID":"f1814684-56ff-4b44-8328-29cbe0497bf2","Type":"ContainerDied","Data":"c21741ff7f845959397e5d29d924dd1521c375ae3477381719c81a89f4471717"} Oct 30 13:15:05.947414 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:05.947401 4931 scope.go:122] "RemoveContainer" containerID="e79b17db4eef990f94670cd9efe3642c428c2d985a304493cca2d814d4a3c3f0" Oct 30 13:15:05.947993 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:05.947868 4931 scope.go:122] "RemoveContainer" containerID="c21741ff7f845959397e5d29d924dd1521c375ae3477381719c81a89f4471717" Oct 30 13:15:05.948326 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:15:05.948242 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-31e10b-29889435-tt6md_odh-ai-gateway-infra(f1814684-56ff-4b44-8328-29cbe0497bf2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31e10b-29889435-tt6md" podUID="f1814684-56ff-4b44-8328-29cbe0497bf2" Oct 30 13:15:05.951828 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:05.951789 4931 generic.go:356] "Generic (PLEG): container finished" podID="f06e3aa4-5df5-45c3-a80e-02b922c62768" containerID="34ba8f6af56ac562bbd3dae00f48a6cafdfe60fadb1b551523729c93d136108f" exitCode=60 Oct 30 13:15:05.951942 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:05.951915 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-h2hfn" event={"ID":"f06e3aa4-5df5-45c3-a80e-02b922c62768","Type":"ContainerDied","Data":"34ba8f6af56ac562bbd3dae00f48a6cafdfe60fadb1b551523729c93d136108f"} Oct 30 13:15:05.952350 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:05.952328 4931 scope.go:122] "RemoveContainer" containerID="34ba8f6af56ac562bbd3dae00f48a6cafdfe60fadb1b551523729c93d136108f" Oct 30 13:15:05.952972 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:15:05.952767 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889435-h2hfn_odh-ai-gateway-infra(f06e3aa4-5df5-45c3-a80e-02b922c62768)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-h2hfn" podUID="f06e3aa4-5df5-45c3-a80e-02b922c62768" Oct 30 13:15:05.956905 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:05.956874 4931 scope.go:122] "RemoveContainer" containerID="064d14caeb9e6b558442ac8ca3b1971bd1d69c54808471b82fbf38f7f6e322f9" Oct 30 13:15:05.959966 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:05.959928 4931 generic.go:356] "Generic (PLEG): container finished" podID="c3bc7915-8ed0-47c5-a693-15db3d153fe8" containerID="8bd26f344bfab5e67d4561e09103026fec443584da08d4db313d91d1baf6f471" exitCode=60 Oct 30 13:15:05.960464 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:05.960340 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj" event={"ID":"c3bc7915-8ed0-47c5-a693-15db3d153fe8","Type":"ContainerDied","Data":"8bd26f344bfab5e67d4561e09103026fec443584da08d4db313d91d1baf6f471"} Oct 30 13:15:05.974787 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:05.974760 4931 generic.go:356] "Generic (PLEG): container finished" podID="0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b" containerID="b2f59eaa6e41ec8282f2c9a235954aebe090e8458d0d4e016faf0d6458a8ec8a" exitCode=60 Oct 30 13:15:05.975225 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:05.974919 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5b9e8e-29889435-6cs4n" event={"ID":"0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b","Type":"ContainerDied","Data":"b2f59eaa6e41ec8282f2c9a235954aebe090e8458d0d4e016faf0d6458a8ec8a"} Oct 30 13:15:05.980327 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:05.975094 4931 scope.go:122] "RemoveContainer" containerID="879cdb4f8d7fff02d514afeae8a196413e02e877048ba98695504b27a3fb902d" Oct 30 13:15:05.980508 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:15:05.975687 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2_odh-ai-gateway-infra(2a5f8729-df67-4e02-b7c5-f2250efa9678)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2" podUID="2a5f8729-df67-4e02-b7c5-f2250efa9678" Oct 30 13:15:05.981178 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:05.980620 4931 scope.go:122] "RemoveContainer" containerID="b2f59eaa6e41ec8282f2c9a235954aebe090e8458d0d4e016faf0d6458a8ec8a" Oct 30 13:15:05.981178 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:05.981104 4931 scope.go:122] "RemoveContainer" containerID="8bd26f344bfab5e67d4561e09103026fec443584da08d4db313d91d1baf6f471" Oct 30 13:15:05.981452 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:15:05.981424 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj_odh-ai-gateway-infra(c3bc7915-8ed0-47c5-a693-15db3d153fe8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj" podUID="c3bc7915-8ed0-47c5-a693-15db3d153fe8" Oct 30 13:15:05.982794 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:15:05.982558 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-5b9e8e-29889435-6cs4n_odh-ai-gateway-infra(0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5b9e8e-29889435-6cs4n" podUID="0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b" Oct 30 13:15:05.985480 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:05.985458 4931 generic.go:356] "Generic (PLEG): container finished" podID="d882efb1-cd4f-45ce-b829-4795003e31b9" containerID="b07113b9d65b418286f2134d75ef4c3fc42a37f8cc01da459ae902069f2ce6a4" exitCode=60 Oct 30 13:15:05.985760 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:05.985739 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-14e8a4-29889435-cldf2" event={"ID":"d882efb1-cd4f-45ce-b829-4795003e31b9","Type":"ContainerDied","Data":"b07113b9d65b418286f2134d75ef4c3fc42a37f8cc01da459ae902069f2ce6a4"} Oct 30 13:15:05.986627 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:05.986608 4931 scope.go:122] "RemoveContainer" containerID="b07113b9d65b418286f2134d75ef4c3fc42a37f8cc01da459ae902069f2ce6a4" Oct 30 13:15:05.987037 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:15:05.987014 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-14e8a4-29889435-cldf2_odh-ai-gateway-infra(d882efb1-cd4f-45ce-b829-4795003e31b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-14e8a4-29889435-cldf2" podUID="d882efb1-cd4f-45ce-b829-4795003e31b9" Oct 30 13:15:05.989989 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:05.989966 4931 generic.go:356] "Generic (PLEG): container finished" podID="ec2e9458-537c-428a-abbe-6ff129af3c6e" containerID="8679f81ed0b750b1b32ca024eb9600ee302b7051fd6c3ce9c5512cde4b99544b" exitCode=60 Oct 30 13:15:05.990196 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:05.990047 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4a8ee1-29889435-5w88z" event={"ID":"ec2e9458-537c-428a-abbe-6ff129af3c6e","Type":"ContainerDied","Data":"8679f81ed0b750b1b32ca024eb9600ee302b7051fd6c3ce9c5512cde4b99544b"} Oct 30 13:15:05.992990 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:05.992967 4931 generic.go:356] "Generic (PLEG): container finished" podID="795f8222-5020-4ef8-be21-49fbb8b178e8" containerID="81a563c30bab377fb31f64628df2799df51e6729a7db27da9700da0525c1f6ea" exitCode=60 Oct 30 13:15:05.993138 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:05.993113 4931 scope.go:122] "RemoveContainer" containerID="8679f81ed0b750b1b32ca024eb9600ee302b7051fd6c3ce9c5512cde4b99544b" Oct 30 13:15:05.993298 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:05.993257 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889435-q4q4t" event={"ID":"795f8222-5020-4ef8-be21-49fbb8b178e8","Type":"ContainerDied","Data":"81a563c30bab377fb31f64628df2799df51e6729a7db27da9700da0525c1f6ea"} Oct 30 13:15:05.993760 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:15:05.993440 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-4a8ee1-29889435-5w88z_odh-ai-gateway-infra(ec2e9458-537c-428a-abbe-6ff129af3c6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4a8ee1-29889435-5w88z" podUID="ec2e9458-537c-428a-abbe-6ff129af3c6e" Oct 30 13:15:05.993760 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:05.993635 4931 scope.go:122] "RemoveContainer" containerID="81a563c30bab377fb31f64628df2799df51e6729a7db27da9700da0525c1f6ea" Oct 30 13:15:05.993924 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:15:05.993892 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889435-q4q4t_odh-ai-gateway-infra(795f8222-5020-4ef8-be21-49fbb8b178e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889435-q4q4t" podUID="795f8222-5020-4ef8-be21-49fbb8b178e8" Oct 30 13:15:05.996322 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:05.996297 4931 generic.go:356] "Generic (PLEG): container finished" podID="08e53b66-2198-4d37-82d5-563f36393d67" containerID="5388bbb1ed5edc19ba11d5ad380d23d471f2dbbce0fdea098f29d4e36a2b9990" exitCode=60 Oct 30 13:15:05.996416 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:05.996395 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv" event={"ID":"08e53b66-2198-4d37-82d5-563f36393d67","Type":"ContainerDied","Data":"5388bbb1ed5edc19ba11d5ad380d23d471f2dbbce0fdea098f29d4e36a2b9990"} Oct 30 13:15:05.996664 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:05.996643 4931 scope.go:122] "RemoveContainer" containerID="5388bbb1ed5edc19ba11d5ad380d23d471f2dbbce0fdea098f29d4e36a2b9990" Oct 30 13:15:05.997083 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:15:05.996899 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv_odh-ai-gateway-infra(08e53b66-2198-4d37-82d5-563f36393d67)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv" podUID="08e53b66-2198-4d37-82d5-563f36393d67" Oct 30 13:15:05.998516 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:05.998489 4931 scope.go:122] "RemoveContainer" containerID="d3bf2e6f3d52109bfbad1217c9cbe9b9d5c4dd6adc5963beaa7aba296969dc55" Oct 30 13:15:05.998691 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:15:05.998671 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh_odh-ai-gateway-infra(01a78c34-1e58-48f3-8660-56d6b19d8847)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh" podUID="01a78c34-1e58-48f3-8660-56d6b19d8847" Oct 30 13:15:06.002845 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:06.002824 4931 scope.go:122] "RemoveContainer" containerID="da5fd92b60a324539bd8653988db5e29ce4e017bae6356ececad22b32ee02599" Oct 30 13:15:06.023692 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:06.023667 4931 scope.go:122] "RemoveContainer" containerID="e1803b357afb6d8d2121a62e2eb228dff1b8e61eb86b01802b257f6962af1b67" Oct 30 13:15:06.041707 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:06.041679 4931 scope.go:122] "RemoveContainer" containerID="15898ffe8fa496be25d803a19667bec970fe86893a715188569c0a2f074e5b65" Oct 30 13:15:06.060723 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:06.060664 4931 scope.go:122] "RemoveContainer" containerID="2f972d748ef051be6bdb94b18c2ecccf7496e0c0b1fe5f8ff44fdd2b88a61f21" Oct 30 13:15:06.082632 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:06.082609 4931 scope.go:122] "RemoveContainer" containerID="abd5554a8570829d92db46b64902475716a3ca59311249a14f5089f3cc24d53d" Oct 30 13:15:06.109989 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:06.108379 4931 scope.go:122] "RemoveContainer" containerID="e7059497e0281af8fd7ae3bbb128532f2477a49509bdad1dce072cc652a9287a" Oct 30 13:15:07.022861 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:07.022827 4931 scope.go:122] "RemoveContainer" containerID="34ba8f6af56ac562bbd3dae00f48a6cafdfe60fadb1b551523729c93d136108f" Oct 30 13:15:07.023344 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:15:07.023219 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889435-h2hfn_odh-ai-gateway-infra(f06e3aa4-5df5-45c3-a80e-02b922c62768)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-h2hfn" podUID="f06e3aa4-5df5-45c3-a80e-02b922c62768" Oct 30 13:15:07.025095 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:07.025073 4931 scope.go:122] "RemoveContainer" containerID="8bd26f344bfab5e67d4561e09103026fec443584da08d4db313d91d1baf6f471" Oct 30 13:15:07.025397 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:15:07.025361 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj_odh-ai-gateway-infra(c3bc7915-8ed0-47c5-a693-15db3d153fe8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj" podUID="c3bc7915-8ed0-47c5-a693-15db3d153fe8" Oct 30 13:15:07.031742 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:07.031610 4931 scope.go:122] "RemoveContainer" containerID="81a563c30bab377fb31f64628df2799df51e6729a7db27da9700da0525c1f6ea" Oct 30 13:15:07.031911 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:15:07.031888 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889435-q4q4t_odh-ai-gateway-infra(795f8222-5020-4ef8-be21-49fbb8b178e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889435-q4q4t" podUID="795f8222-5020-4ef8-be21-49fbb8b178e8" Oct 30 13:15:07.049313 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:07.047128 4931 scope.go:122] "RemoveContainer" containerID="5388bbb1ed5edc19ba11d5ad380d23d471f2dbbce0fdea098f29d4e36a2b9990" Oct 30 13:15:07.049313 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:15:07.047672 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv_odh-ai-gateway-infra(08e53b66-2198-4d37-82d5-563f36393d67)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv" podUID="08e53b66-2198-4d37-82d5-563f36393d67" Oct 30 13:15:07.615289 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:15:07.608864 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:15:12.399774 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:12.399728 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6"] Oct 30 13:15:12.400287 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:12.399977 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" podUID="29a14a0f-98aa-4981-abe4-56919051950b" containerName="istio-proxy" containerID="cri-o://18c18901545c45e7aff743ac7204a95e8d926847e5e6e82f30222c81deadd3a4" gracePeriod=30 Oct 30 13:15:14.506186 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:14.506157 4931 scope.go:122] "RemoveContainer" containerID="4cb9280b9412719c415c6732759e3cfe1f6472e49a992928646cf269616ffc76" Oct 30 13:15:15.210701 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:15.210668 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj" event={"ID":"ea3c064a-c226-4eec-bd7e-d42fb03965ad","Type":"ContainerStarted","Data":"9dc6e7b894d494943fc9c5bcb76343113bafa8893e951444688520992f86079c"} Oct 30 13:15:16.232527 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:16.232476 4931 generic.go:356] "Generic (PLEG): container finished" podID="ea3c064a-c226-4eec-bd7e-d42fb03965ad" containerID="9dc6e7b894d494943fc9c5bcb76343113bafa8893e951444688520992f86079c" exitCode=60 Oct 30 13:15:16.233145 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:16.232658 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj" event={"ID":"ea3c064a-c226-4eec-bd7e-d42fb03965ad","Type":"ContainerDied","Data":"9dc6e7b894d494943fc9c5bcb76343113bafa8893e951444688520992f86079c"} Oct 30 13:15:16.233145 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:16.232761 4931 scope.go:122] "RemoveContainer" containerID="4cb9280b9412719c415c6732759e3cfe1f6472e49a992928646cf269616ffc76" Oct 30 13:15:16.233340 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:16.233321 4931 scope.go:122] "RemoveContainer" containerID="9dc6e7b894d494943fc9c5bcb76343113bafa8893e951444688520992f86079c" Oct 30 13:15:16.233866 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:15:16.233823 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj_odh-ai-gateway-infra(ea3c064a-c226-4eec-bd7e-d42fb03965ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj" podUID="ea3c064a-c226-4eec-bd7e-d42fb03965ad" Oct 30 13:15:16.269469 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:16.269432 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj"] Oct 30 13:15:16.506722 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:16.506636 4931 scope.go:122] "RemoveContainer" containerID="b2f59eaa6e41ec8282f2c9a235954aebe090e8458d0d4e016faf0d6458a8ec8a" Oct 30 13:15:16.506882 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:16.506831 4931 scope.go:122] "RemoveContainer" containerID="c21741ff7f845959397e5d29d924dd1521c375ae3477381719c81a89f4471717" Oct 30 13:15:16.507036 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:16.507006 4931 scope.go:122] "RemoveContainer" containerID="5388bbb1ed5edc19ba11d5ad380d23d471f2dbbce0fdea098f29d4e36a2b9990" Oct 30 13:15:17.245349 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:17.245309 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5b9e8e-29889435-6cs4n" event={"ID":"0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b","Type":"ContainerStarted","Data":"107e00612f7feb62db07fb3d668d692f7cf135479f26d8a197f43882878b3150"} Oct 30 13:15:17.247767 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:17.247739 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv" event={"ID":"08e53b66-2198-4d37-82d5-563f36393d67","Type":"ContainerStarted","Data":"5f7dfaef4d0ef19af92e63d9c6d24c19f6c93798468450bdd0111b5af235c5bd"} Oct 30 13:15:17.251929 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:17.251907 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31e10b-29889435-tt6md" event={"ID":"f1814684-56ff-4b44-8328-29cbe0497bf2","Type":"ContainerStarted","Data":"d9771dab73c7bea02ea6a0882cbf85a6dc1e7b2dec96e334814dc9bf2c353c35"} Oct 30 13:15:17.503938 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:17.503907 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj" Oct 30 13:15:17.506468 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:17.506445 4931 scope.go:122] "RemoveContainer" containerID="064d14caeb9e6b558442ac8ca3b1971bd1d69c54808471b82fbf38f7f6e322f9" Oct 30 13:15:17.568112 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:17.568072 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31e10b-29889435-tt6md"] Oct 30 13:15:17.615151 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:17.615107 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5b9e8e-29889435-6cs4n"] Oct 30 13:15:17.660013 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:17.659928 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea3c064a-c226-4eec-bd7e-d42fb03965ad-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea3c064a-c226-4eec-bd7e-d42fb03965ad-maas-api-service-ca\") pod \"ea3c064a-c226-4eec-bd7e-d42fb03965ad\" (UID: \"ea3c064a-c226-4eec-bd7e-d42fb03965ad\") " Oct 30 13:15:17.660181 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:17.660103 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea3c064a-c226-4eec-bd7e-d42fb03965ad-kube-api-access-vh7v9\" (UniqueName: \"kubernetes.io/projected/ea3c064a-c226-4eec-bd7e-d42fb03965ad-kube-api-access-vh7v9\") pod \"ea3c064a-c226-4eec-bd7e-d42fb03965ad\" (UID: \"ea3c064a-c226-4eec-bd7e-d42fb03965ad\") " Oct 30 13:15:17.660720 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:17.660643 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea3c064a-c226-4eec-bd7e-d42fb03965ad-maas-api-service-ca" pod "ea3c064a-c226-4eec-bd7e-d42fb03965ad" (UID: "ea3c064a-c226-4eec-bd7e-d42fb03965ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:15:17.662097 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:17.662064 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea3c064a-c226-4eec-bd7e-d42fb03965ad-maas-api-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:15:17.663631 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:17.663589 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea3c064a-c226-4eec-bd7e-d42fb03965ad-kube-api-access-vh7v9" pod "ea3c064a-c226-4eec-bd7e-d42fb03965ad" (UID: "ea3c064a-c226-4eec-bd7e-d42fb03965ad"). InnerVolumeSpecName "kube-api-access-vh7v9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:17.764774 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:17.764719 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh7v9\" (UniqueName: \"kubernetes.io/projected/ea3c064a-c226-4eec-bd7e-d42fb03965ad-kube-api-access-vh7v9\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:15:17.958664 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:17.958636 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" Oct 30 13:15:18.068131 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.068100 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29a14a0f-98aa-4981-abe4-56919051950b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29a14a0f-98aa-4981-abe4-56919051950b-istiod-ca-cert\") pod \"29a14a0f-98aa-4981-abe4-56919051950b\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " Oct 30 13:15:18.068402 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.068243 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29a14a0f-98aa-4981-abe4-56919051950b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a14a0f-98aa-4981-abe4-56919051950b-credential-socket\") pod \"29a14a0f-98aa-4981-abe4-56919051950b\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " Oct 30 13:15:18.068402 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.068336 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29a14a0f-98aa-4981-abe4-56919051950b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29a14a0f-98aa-4981-abe4-56919051950b-istio-data\") pod \"29a14a0f-98aa-4981-abe4-56919051950b\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " Oct 30 13:15:18.068402 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.068360 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29a14a0f-98aa-4981-abe4-56919051950b-kube-api-access-rqfnl\" (UniqueName: \"kubernetes.io/projected/29a14a0f-98aa-4981-abe4-56919051950b-kube-api-access-rqfnl\") pod \"29a14a0f-98aa-4981-abe4-56919051950b\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " Oct 30 13:15:18.068402 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.068388 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29a14a0f-98aa-4981-abe4-56919051950b-istio-token\" (UniqueName: \"kubernetes.io/projected/29a14a0f-98aa-4981-abe4-56919051950b-istio-token\") pod \"29a14a0f-98aa-4981-abe4-56919051950b\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " Oct 30 13:15:18.068646 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.068533 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29a14a0f-98aa-4981-abe4-56919051950b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29a14a0f-98aa-4981-abe4-56919051950b-istio-envoy\") pod \"29a14a0f-98aa-4981-abe4-56919051950b\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " Oct 30 13:15:18.068646 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.068578 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29a14a0f-98aa-4981-abe4-56919051950b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29a14a0f-98aa-4981-abe4-56919051950b-istio-podinfo\") pod \"29a14a0f-98aa-4981-abe4-56919051950b\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " Oct 30 13:15:18.068646 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.068580 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a14a0f-98aa-4981-abe4-56919051950b-credential-socket" pod "29a14a0f-98aa-4981-abe4-56919051950b" (UID: "29a14a0f-98aa-4981-abe4-56919051950b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:15:18.068646 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.068612 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29a14a0f-98aa-4981-abe4-56919051950b-istiod-ca-cert" pod "29a14a0f-98aa-4981-abe4-56919051950b" (UID: "29a14a0f-98aa-4981-abe4-56919051950b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:15:18.068646 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.068623 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29a14a0f-98aa-4981-abe4-56919051950b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29a14a0f-98aa-4981-abe4-56919051950b-workload-certs\") pod \"29a14a0f-98aa-4981-abe4-56919051950b\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " Oct 30 13:15:18.068885 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.068658 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a14a0f-98aa-4981-abe4-56919051950b-istio-data" pod "29a14a0f-98aa-4981-abe4-56919051950b" (UID: "29a14a0f-98aa-4981-abe4-56919051950b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:15:18.068885 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.068710 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29a14a0f-98aa-4981-abe4-56919051950b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a14a0f-98aa-4981-abe4-56919051950b-workload-socket\") pod \"29a14a0f-98aa-4981-abe4-56919051950b\" (UID: \"29a14a0f-98aa-4981-abe4-56919051950b\") " Oct 30 13:15:18.068885 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.068827 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a14a0f-98aa-4981-abe4-56919051950b-workload-certs" pod "29a14a0f-98aa-4981-abe4-56919051950b" (UID: "29a14a0f-98aa-4981-abe4-56919051950b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:15:18.069123 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.069098 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a14a0f-98aa-4981-abe4-56919051950b-workload-socket" pod "29a14a0f-98aa-4981-abe4-56919051950b" (UID: "29a14a0f-98aa-4981-abe4-56919051950b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:15:18.069558 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.069532 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29a14a0f-98aa-4981-abe4-56919051950b-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:15:18.069683 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.069562 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a14a0f-98aa-4981-abe4-56919051950b-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:15:18.069683 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.069576 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29a14a0f-98aa-4981-abe4-56919051950b-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:15:18.069683 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.069590 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a14a0f-98aa-4981-abe4-56919051950b-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:15:18.069683 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.069604 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29a14a0f-98aa-4981-abe4-56919051950b-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:15:18.070881 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.070857 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a14a0f-98aa-4981-abe4-56919051950b-istio-token" pod "29a14a0f-98aa-4981-abe4-56919051950b" (UID: "29a14a0f-98aa-4981-abe4-56919051950b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:18.070987 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.070889 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29a14a0f-98aa-4981-abe4-56919051950b-istio-podinfo" pod "29a14a0f-98aa-4981-abe4-56919051950b" (UID: "29a14a0f-98aa-4981-abe4-56919051950b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:15:18.071139 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.071119 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a14a0f-98aa-4981-abe4-56919051950b-kube-api-access-rqfnl" pod "29a14a0f-98aa-4981-abe4-56919051950b" (UID: "29a14a0f-98aa-4981-abe4-56919051950b"). InnerVolumeSpecName "kube-api-access-rqfnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:18.071181 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.071125 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a14a0f-98aa-4981-abe4-56919051950b-istio-envoy" pod "29a14a0f-98aa-4981-abe4-56919051950b" (UID: "29a14a0f-98aa-4981-abe4-56919051950b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:15:18.171055 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.171017 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqfnl\" (UniqueName: \"kubernetes.io/projected/29a14a0f-98aa-4981-abe4-56919051950b-kube-api-access-rqfnl\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:15:18.171055 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.171053 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29a14a0f-98aa-4981-abe4-56919051950b-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:15:18.171055 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.171063 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29a14a0f-98aa-4981-abe4-56919051950b-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:15:18.171335 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.171071 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29a14a0f-98aa-4981-abe4-56919051950b-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:15:18.261941 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.261849 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2" event={"ID":"2a5f8729-df67-4e02-b7c5-f2250efa9678","Type":"ContainerStarted","Data":"d25a4623fc82f949dd783fe31a56a31af9957f2c5ad87524f6d79a6e7b08ae78"} Oct 30 13:15:18.263901 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.263874 4931 generic.go:356] "Generic (PLEG): container finished" podID="0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b" containerID="107e00612f7feb62db07fb3d668d692f7cf135479f26d8a197f43882878b3150" exitCode=60 Oct 30 13:15:18.264023 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.263947 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5b9e8e-29889435-6cs4n" event={"ID":"0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b","Type":"ContainerDied","Data":"107e00612f7feb62db07fb3d668d692f7cf135479f26d8a197f43882878b3150"} Oct 30 13:15:18.264023 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.263995 4931 scope.go:122] "RemoveContainer" containerID="b2f59eaa6e41ec8282f2c9a235954aebe090e8458d0d4e016faf0d6458a8ec8a" Oct 30 13:15:18.265622 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.265598 4931 generic.go:356] "Generic (PLEG): container finished" podID="29a14a0f-98aa-4981-abe4-56919051950b" containerID="18c18901545c45e7aff743ac7204a95e8d926847e5e6e82f30222c81deadd3a4" exitCode=0 Oct 30 13:15:18.265728 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.265632 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" Oct 30 13:15:18.265728 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.265675 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" event={"ID":"29a14a0f-98aa-4981-abe4-56919051950b","Type":"ContainerDied","Data":"18c18901545c45e7aff743ac7204a95e8d926847e5e6e82f30222c81deadd3a4"} Oct 30 13:15:18.265728 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.265717 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6" event={"ID":"29a14a0f-98aa-4981-abe4-56919051950b","Type":"ContainerDied","Data":"a5b5192d4486af132a3562fe9624516d6742b1bae928f9594744baeb05340f6c"} Oct 30 13:15:18.267955 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.267934 4931 generic.go:356] "Generic (PLEG): container finished" podID="08e53b66-2198-4d37-82d5-563f36393d67" containerID="5f7dfaef4d0ef19af92e63d9c6d24c19f6c93798468450bdd0111b5af235c5bd" exitCode=60 Oct 30 13:15:18.268037 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.268006 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv" event={"ID":"08e53b66-2198-4d37-82d5-563f36393d67","Type":"ContainerDied","Data":"5f7dfaef4d0ef19af92e63d9c6d24c19f6c93798468450bdd0111b5af235c5bd"} Oct 30 13:15:18.268465 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.268442 4931 scope.go:122] "RemoveContainer" containerID="5f7dfaef4d0ef19af92e63d9c6d24c19f6c93798468450bdd0111b5af235c5bd" Oct 30 13:15:18.268819 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:15:18.268795 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv_odh-ai-gateway-infra(08e53b66-2198-4d37-82d5-563f36393d67)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv" podUID="08e53b66-2198-4d37-82d5-563f36393d67" Oct 30 13:15:18.269744 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.269726 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj" Oct 30 13:15:18.269842 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.269728 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj" event={"ID":"ea3c064a-c226-4eec-bd7e-d42fb03965ad","Type":"ContainerDied","Data":"d5983de0b3f97ee8a05e3a431d8ef8ef5189200ad88d94109d4fa78607378fd4"} Oct 30 13:15:18.273168 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.273145 4931 generic.go:356] "Generic (PLEG): container finished" podID="f1814684-56ff-4b44-8328-29cbe0497bf2" containerID="d9771dab73c7bea02ea6a0882cbf85a6dc1e7b2dec96e334814dc9bf2c353c35" exitCode=60 Oct 30 13:15:18.273324 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.273181 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31e10b-29889435-tt6md" event={"ID":"f1814684-56ff-4b44-8328-29cbe0497bf2","Type":"ContainerDied","Data":"d9771dab73c7bea02ea6a0882cbf85a6dc1e7b2dec96e334814dc9bf2c353c35"} Oct 30 13:15:18.286944 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.286772 4931 scope.go:122] "RemoveContainer" containerID="18c18901545c45e7aff743ac7204a95e8d926847e5e6e82f30222c81deadd3a4" Oct 30 13:15:18.302803 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.302770 4931 scope.go:122] "RemoveContainer" containerID="18c18901545c45e7aff743ac7204a95e8d926847e5e6e82f30222c81deadd3a4" Oct 30 13:15:18.303230 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:15:18.303200 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18c18901545c45e7aff743ac7204a95e8d926847e5e6e82f30222c81deadd3a4\": container with ID starting with 18c18901545c45e7aff743ac7204a95e8d926847e5e6e82f30222c81deadd3a4 not found: ID does not exist" containerID="18c18901545c45e7aff743ac7204a95e8d926847e5e6e82f30222c81deadd3a4" Oct 30 13:15:18.303347 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.303244 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18c18901545c45e7aff743ac7204a95e8d926847e5e6e82f30222c81deadd3a4"} err="failed to get container status \"18c18901545c45e7aff743ac7204a95e8d926847e5e6e82f30222c81deadd3a4\": rpc error: code = NotFound desc = could not find container \"18c18901545c45e7aff743ac7204a95e8d926847e5e6e82f30222c81deadd3a4\": container with ID starting with 18c18901545c45e7aff743ac7204a95e8d926847e5e6e82f30222c81deadd3a4 not found: ID does not exist" Oct 30 13:15:18.303347 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.303318 4931 scope.go:122] "RemoveContainer" containerID="5388bbb1ed5edc19ba11d5ad380d23d471f2dbbce0fdea098f29d4e36a2b9990" Oct 30 13:15:18.325858 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.325837 4931 scope.go:122] "RemoveContainer" containerID="9dc6e7b894d494943fc9c5bcb76343113bafa8893e951444688520992f86079c" Oct 30 13:15:18.338370 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.338084 4931 scope.go:122] "RemoveContainer" containerID="c21741ff7f845959397e5d29d924dd1521c375ae3477381719c81a89f4471717" Oct 30 13:15:18.434708 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.434674 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv"] Oct 30 13:15:18.463785 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.463749 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5b9e8e-29889435-6cs4n" Oct 30 13:15:18.469674 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.469627 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj"] Oct 30 13:15:18.484912 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.484877 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889435-hx6nj"] Oct 30 13:15:18.497312 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.497280 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6"] Oct 30 13:15:18.504200 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.504180 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31e10b-29889435-tt6md" Oct 30 13:15:18.505416 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.505396 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6"] Oct 30 13:15:18.506189 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.506170 4931 scope.go:122] "RemoveContainer" containerID="d3bf2e6f3d52109bfbad1217c9cbe9b9d5c4dd6adc5963beaa7aba296969dc55" Oct 30 13:15:18.506330 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.506316 4931 scope.go:122] "RemoveContainer" containerID="8bd26f344bfab5e67d4561e09103026fec443584da08d4db313d91d1baf6f471" Oct 30 13:15:18.506401 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.506385 4931 scope.go:122] "RemoveContainer" containerID="b07113b9d65b418286f2134d75ef4c3fc42a37f8cc01da459ae902069f2ce6a4" Oct 30 13:15:18.577547 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.577515 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b-kube-api-access-5kcll\" (UniqueName: \"kubernetes.io/projected/0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b-kube-api-access-5kcll\") pod \"0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b\" (UID: \"0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b\") " Oct 30 13:15:18.577732 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.577715 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b-maas-api-service-ca\") pod \"0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b\" (UID: \"0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b\") " Oct 30 13:15:18.577792 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.577750 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1814684-56ff-4b44-8328-29cbe0497bf2-kube-api-access-khkhv\" (UniqueName: \"kubernetes.io/projected/f1814684-56ff-4b44-8328-29cbe0497bf2-kube-api-access-khkhv\") pod \"f1814684-56ff-4b44-8328-29cbe0497bf2\" (UID: \"f1814684-56ff-4b44-8328-29cbe0497bf2\") " Oct 30 13:15:18.577841 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.577825 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1814684-56ff-4b44-8328-29cbe0497bf2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1814684-56ff-4b44-8328-29cbe0497bf2-maas-api-service-ca\") pod \"f1814684-56ff-4b44-8328-29cbe0497bf2\" (UID: \"f1814684-56ff-4b44-8328-29cbe0497bf2\") " Oct 30 13:15:18.578180 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.578128 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b-maas-api-service-ca" pod "0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b" (UID: "0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:15:18.578304 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.578241 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1814684-56ff-4b44-8328-29cbe0497bf2-maas-api-service-ca" pod "f1814684-56ff-4b44-8328-29cbe0497bf2" (UID: "f1814684-56ff-4b44-8328-29cbe0497bf2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:15:18.578889 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.578854 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b-maas-api-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:15:18.578960 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.578896 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1814684-56ff-4b44-8328-29cbe0497bf2-maas-api-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:15:18.580125 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.580099 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1814684-56ff-4b44-8328-29cbe0497bf2-kube-api-access-khkhv" pod "f1814684-56ff-4b44-8328-29cbe0497bf2" (UID: "f1814684-56ff-4b44-8328-29cbe0497bf2"). InnerVolumeSpecName "kube-api-access-khkhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:18.580220 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.580128 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b-kube-api-access-5kcll" pod "0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b" (UID: "0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b"). InnerVolumeSpecName "kube-api-access-5kcll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:18.680527 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.680491 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kcll\" (UniqueName: \"kubernetes.io/projected/0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b-kube-api-access-5kcll\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:15:18.680661 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:18.680531 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khkhv\" (UniqueName: \"kubernetes.io/projected/f1814684-56ff-4b44-8328-29cbe0497bf2-kube-api-access-khkhv\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:15:19.284188 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:19.284153 4931 generic.go:356] "Generic (PLEG): container finished" podID="2a5f8729-df67-4e02-b7c5-f2250efa9678" containerID="d25a4623fc82f949dd783fe31a56a31af9957f2c5ad87524f6d79a6e7b08ae78" exitCode=60 Oct 30 13:15:19.285410 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:19.284225 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2" event={"ID":"2a5f8729-df67-4e02-b7c5-f2250efa9678","Type":"ContainerDied","Data":"d25a4623fc82f949dd783fe31a56a31af9957f2c5ad87524f6d79a6e7b08ae78"} Oct 30 13:15:19.285410 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:19.284331 4931 scope.go:122] "RemoveContainer" containerID="064d14caeb9e6b558442ac8ca3b1971bd1d69c54808471b82fbf38f7f6e322f9" Oct 30 13:15:19.285410 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:19.284813 4931 scope.go:122] "RemoveContainer" containerID="d25a4623fc82f949dd783fe31a56a31af9957f2c5ad87524f6d79a6e7b08ae78" Oct 30 13:15:19.285410 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:15:19.285250 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2_odh-ai-gateway-infra(2a5f8729-df67-4e02-b7c5-f2250efa9678)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2" podUID="2a5f8729-df67-4e02-b7c5-f2250efa9678" Oct 30 13:15:19.286538 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:19.286516 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj" event={"ID":"c3bc7915-8ed0-47c5-a693-15db3d153fe8","Type":"ContainerStarted","Data":"5b9ef4093290232d4ebf4ec3cde5f426dcbfd6d8fbf67feb7251fd809b5c15a3"} Oct 30 13:15:19.287809 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:19.287791 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5b9e8e-29889435-6cs4n" Oct 30 13:15:19.287893 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:19.287828 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5b9e8e-29889435-6cs4n" event={"ID":"0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b","Type":"ContainerDied","Data":"b0cbadd3a22a8cdd6f060a7e6ee178260ba00be51f0a95491fbc5a50b44a264a"} Oct 30 13:15:19.289878 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:19.289853 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-14e8a4-29889435-cldf2" event={"ID":"d882efb1-cd4f-45ce-b829-4795003e31b9","Type":"ContainerStarted","Data":"a0b5b2e700f9e85b64f45f656b57b5f4e952c92a6ad02314b821d0aa9f07785e"} Oct 30 13:15:19.295575 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:19.295546 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh" event={"ID":"01a78c34-1e58-48f3-8660-56d6b19d8847","Type":"ContainerStarted","Data":"46d2fd777f87a2996dc63d919032f2f639294d7fe2b69eb497a6ddf6e42184ad"} Oct 30 13:15:19.298364 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:19.298341 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31e10b-29889435-tt6md" event={"ID":"f1814684-56ff-4b44-8328-29cbe0497bf2","Type":"ContainerDied","Data":"33735a3e149e4a6069dd9ed1df627b815ddcda02475bb43e8cbeb1db50c59800"} Oct 30 13:15:19.298470 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:19.298391 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31e10b-29889435-tt6md" Oct 30 13:15:19.300735 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:19.300707 4931 scope.go:122] "RemoveContainer" containerID="107e00612f7feb62db07fb3d668d692f7cf135479f26d8a197f43882878b3150" Oct 30 13:15:19.376100 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:19.376073 4931 scope.go:122] "RemoveContainer" containerID="d9771dab73c7bea02ea6a0882cbf85a6dc1e7b2dec96e334814dc9bf2c353c35" Oct 30 13:15:19.379834 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:19.379803 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2"] Oct 30 13:15:19.431678 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:19.431573 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31e10b-29889435-tt6md"] Oct 30 13:15:19.444384 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:19.443729 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31e10b-29889435-tt6md"] Oct 30 13:15:19.479550 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:19.479511 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5b9e8e-29889435-6cs4n"] Oct 30 13:15:19.484087 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:19.484058 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5b9e8e-29889435-6cs4n"] Oct 30 13:15:19.516653 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:19.516615 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b" path="/var/lib/kubelet/pods/0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b/volumes" Oct 30 13:15:19.517253 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:19.517233 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29a14a0f-98aa-4981-abe4-56919051950b" path="/var/lib/kubelet/pods/29a14a0f-98aa-4981-abe4-56919051950b/volumes" Oct 30 13:15:19.517852 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:19.517829 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea3c064a-c226-4eec-bd7e-d42fb03965ad" path="/var/lib/kubelet/pods/ea3c064a-c226-4eec-bd7e-d42fb03965ad/volumes" Oct 30 13:15:19.518869 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:19.518847 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1814684-56ff-4b44-8328-29cbe0497bf2" path="/var/lib/kubelet/pods/f1814684-56ff-4b44-8328-29cbe0497bf2/volumes" Oct 30 13:15:19.520237 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:19.520216 4931 scope.go:122] "RemoveContainer" containerID="81a563c30bab377fb31f64628df2799df51e6729a7db27da9700da0525c1f6ea" Oct 30 13:15:19.520420 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:19.520401 4931 scope.go:122] "RemoveContainer" containerID="8679f81ed0b750b1b32ca024eb9600ee302b7051fd6c3ce9c5512cde4b99544b" Oct 30 13:15:19.520547 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:19.520528 4931 scope.go:122] "RemoveContainer" containerID="34ba8f6af56ac562bbd3dae00f48a6cafdfe60fadb1b551523729c93d136108f" Oct 30 13:15:19.543919 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:19.541508 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv" Oct 30 13:15:19.601689 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:19.601658 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-14e8a4-29889435-cldf2"] Oct 30 13:15:19.698459 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:19.698419 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08e53b66-2198-4d37-82d5-563f36393d67-kube-api-access-rtj5s\" (UniqueName: \"kubernetes.io/projected/08e53b66-2198-4d37-82d5-563f36393d67-kube-api-access-rtj5s\") pod \"08e53b66-2198-4d37-82d5-563f36393d67\" (UID: \"08e53b66-2198-4d37-82d5-563f36393d67\") " Oct 30 13:15:19.698682 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:19.698660 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08e53b66-2198-4d37-82d5-563f36393d67-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08e53b66-2198-4d37-82d5-563f36393d67-maas-api-service-ca\") pod \"08e53b66-2198-4d37-82d5-563f36393d67\" (UID: \"08e53b66-2198-4d37-82d5-563f36393d67\") " Oct 30 13:15:19.700190 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:19.700154 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08e53b66-2198-4d37-82d5-563f36393d67-maas-api-service-ca" pod "08e53b66-2198-4d37-82d5-563f36393d67" (UID: "08e53b66-2198-4d37-82d5-563f36393d67"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:15:19.742950 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:19.742886 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08e53b66-2198-4d37-82d5-563f36393d67-kube-api-access-rtj5s" pod "08e53b66-2198-4d37-82d5-563f36393d67" (UID: "08e53b66-2198-4d37-82d5-563f36393d67"). InnerVolumeSpecName "kube-api-access-rtj5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:19.829093 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:19.824651 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08e53b66-2198-4d37-82d5-563f36393d67-maas-api-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:15:19.829093 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:19.824689 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtj5s\" (UniqueName: \"kubernetes.io/projected/08e53b66-2198-4d37-82d5-563f36393d67-kube-api-access-rtj5s\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:15:20.311572 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.311540 4931 generic.go:356] "Generic (PLEG): container finished" podID="c3bc7915-8ed0-47c5-a693-15db3d153fe8" containerID="5b9ef4093290232d4ebf4ec3cde5f426dcbfd6d8fbf67feb7251fd809b5c15a3" exitCode=60 Oct 30 13:15:20.311980 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.311618 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj" event={"ID":"c3bc7915-8ed0-47c5-a693-15db3d153fe8","Type":"ContainerDied","Data":"5b9ef4093290232d4ebf4ec3cde5f426dcbfd6d8fbf67feb7251fd809b5c15a3"} Oct 30 13:15:20.311980 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.311727 4931 scope.go:122] "RemoveContainer" containerID="8bd26f344bfab5e67d4561e09103026fec443584da08d4db313d91d1baf6f471" Oct 30 13:15:20.312251 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.312184 4931 scope.go:122] "RemoveContainer" containerID="5b9ef4093290232d4ebf4ec3cde5f426dcbfd6d8fbf67feb7251fd809b5c15a3" Oct 30 13:15:20.312654 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:15:20.312630 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj_odh-ai-gateway-infra(c3bc7915-8ed0-47c5-a693-15db3d153fe8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj" podUID="c3bc7915-8ed0-47c5-a693-15db3d153fe8" Oct 30 13:15:20.316602 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.316576 4931 generic.go:356] "Generic (PLEG): container finished" podID="d882efb1-cd4f-45ce-b829-4795003e31b9" containerID="a0b5b2e700f9e85b64f45f656b57b5f4e952c92a6ad02314b821d0aa9f07785e" exitCode=60 Oct 30 13:15:20.316724 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.316659 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-14e8a4-29889435-cldf2" event={"ID":"d882efb1-cd4f-45ce-b829-4795003e31b9","Type":"ContainerDied","Data":"a0b5b2e700f9e85b64f45f656b57b5f4e952c92a6ad02314b821d0aa9f07785e"} Oct 30 13:15:20.319419 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.319390 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4a8ee1-29889435-5w88z" event={"ID":"ec2e9458-537c-428a-abbe-6ff129af3c6e","Type":"ContainerStarted","Data":"a90b35c7329f1ea86afd646e100e7a08ff7ce2cbc4f6ca0bc62170679da272b5"} Oct 30 13:15:20.321952 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.321916 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889435-q4q4t" event={"ID":"795f8222-5020-4ef8-be21-49fbb8b178e8","Type":"ContainerStarted","Data":"d3ed2be84e41e8df396d109efb2dd2a4653de759cdefa5602c9b2497add05b0e"} Oct 30 13:15:20.325932 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.324009 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv" Oct 30 13:15:20.325932 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.324026 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv" event={"ID":"08e53b66-2198-4d37-82d5-563f36393d67","Type":"ContainerDied","Data":"5f3b834b341aadef5ca28b0322aea7db2f62c71268fcbd54e423d6884b4629d4"} Oct 30 13:15:20.331323 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.331245 4931 scope.go:122] "RemoveContainer" containerID="b07113b9d65b418286f2134d75ef4c3fc42a37f8cc01da459ae902069f2ce6a4" Oct 30 13:15:20.332205 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.332175 4931 generic.go:356] "Generic (PLEG): container finished" podID="01a78c34-1e58-48f3-8660-56d6b19d8847" containerID="46d2fd777f87a2996dc63d919032f2f639294d7fe2b69eb497a6ddf6e42184ad" exitCode=60 Oct 30 13:15:20.333191 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.333163 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh" event={"ID":"01a78c34-1e58-48f3-8660-56d6b19d8847","Type":"ContainerDied","Data":"46d2fd777f87a2996dc63d919032f2f639294d7fe2b69eb497a6ddf6e42184ad"} Oct 30 13:15:20.334079 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.334061 4931 scope.go:122] "RemoveContainer" containerID="46d2fd777f87a2996dc63d919032f2f639294d7fe2b69eb497a6ddf6e42184ad" Oct 30 13:15:20.334647 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:15:20.334622 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh_odh-ai-gateway-infra(01a78c34-1e58-48f3-8660-56d6b19d8847)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh" podUID="01a78c34-1e58-48f3-8660-56d6b19d8847" Oct 30 13:15:20.344199 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.344169 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-h2hfn" event={"ID":"f06e3aa4-5df5-45c3-a80e-02b922c62768","Type":"ContainerStarted","Data":"f87a0ef7626b712567a9e9e380f386322a9b9425bf9ce5a37ed9306dca6bb7fd"} Oct 30 13:15:20.358458 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.355969 4931 scope.go:122] "RemoveContainer" containerID="5f7dfaef4d0ef19af92e63d9c6d24c19f6c93798468450bdd0111b5af235c5bd" Oct 30 13:15:20.391065 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.390677 4931 scope.go:122] "RemoveContainer" containerID="d3bf2e6f3d52109bfbad1217c9cbe9b9d5c4dd6adc5963beaa7aba296969dc55" Oct 30 13:15:20.533017 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.532982 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj"] Oct 30 13:15:20.547679 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.547646 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2" Oct 30 13:15:20.584965 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.584934 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh"] Oct 30 13:15:20.588011 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.587986 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-14e8a4-29889435-cldf2" Oct 30 13:15:20.646434 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.646403 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a5f8729-df67-4e02-b7c5-f2250efa9678-kube-api-access-2ggzk\" (UniqueName: \"kubernetes.io/projected/2a5f8729-df67-4e02-b7c5-f2250efa9678-kube-api-access-2ggzk\") pod \"2a5f8729-df67-4e02-b7c5-f2250efa9678\" (UID: \"2a5f8729-df67-4e02-b7c5-f2250efa9678\") " Oct 30 13:15:20.646547 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.646522 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a5f8729-df67-4e02-b7c5-f2250efa9678-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a5f8729-df67-4e02-b7c5-f2250efa9678-maas-api-service-ca\") pod \"2a5f8729-df67-4e02-b7c5-f2250efa9678\" (UID: \"2a5f8729-df67-4e02-b7c5-f2250efa9678\") " Oct 30 13:15:20.646872 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.646846 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a5f8729-df67-4e02-b7c5-f2250efa9678-maas-api-service-ca" pod "2a5f8729-df67-4e02-b7c5-f2250efa9678" (UID: "2a5f8729-df67-4e02-b7c5-f2250efa9678"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:15:20.647186 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.647163 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a5f8729-df67-4e02-b7c5-f2250efa9678-maas-api-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:15:20.649124 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.649102 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a5f8729-df67-4e02-b7c5-f2250efa9678-kube-api-access-2ggzk" pod "2a5f8729-df67-4e02-b7c5-f2250efa9678" (UID: "2a5f8729-df67-4e02-b7c5-f2250efa9678"). InnerVolumeSpecName "kube-api-access-2ggzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:20.723096 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.722980 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4a8ee1-29889435-5w88z"] Oct 30 13:15:20.748153 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.748124 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d882efb1-cd4f-45ce-b829-4795003e31b9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d882efb1-cd4f-45ce-b829-4795003e31b9-maas-api-service-ca\") pod \"d882efb1-cd4f-45ce-b829-4795003e31b9\" (UID: \"d882efb1-cd4f-45ce-b829-4795003e31b9\") " Oct 30 13:15:20.748356 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.748341 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d882efb1-cd4f-45ce-b829-4795003e31b9-kube-api-access-cpdtt\" (UniqueName: \"kubernetes.io/projected/d882efb1-cd4f-45ce-b829-4795003e31b9-kube-api-access-cpdtt\") pod \"d882efb1-cd4f-45ce-b829-4795003e31b9\" (UID: \"d882efb1-cd4f-45ce-b829-4795003e31b9\") " Oct 30 13:15:20.748521 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.748497 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d882efb1-cd4f-45ce-b829-4795003e31b9-maas-api-service-ca" pod "d882efb1-cd4f-45ce-b829-4795003e31b9" (UID: "d882efb1-cd4f-45ce-b829-4795003e31b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:15:20.749124 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.749106 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d882efb1-cd4f-45ce-b829-4795003e31b9-maas-api-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:15:20.749167 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.749128 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ggzk\" (UniqueName: \"kubernetes.io/projected/2a5f8729-df67-4e02-b7c5-f2250efa9678-kube-api-access-2ggzk\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:15:20.750533 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.750513 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d882efb1-cd4f-45ce-b829-4795003e31b9-kube-api-access-cpdtt" pod "d882efb1-cd4f-45ce-b829-4795003e31b9" (UID: "d882efb1-cd4f-45ce-b829-4795003e31b9"). InnerVolumeSpecName "kube-api-access-cpdtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:20.850710 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.850617 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpdtt\" (UniqueName: \"kubernetes.io/projected/d882efb1-cd4f-45ce-b829-4795003e31b9-kube-api-access-cpdtt\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:15:20.912568 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.912530 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv"] Oct 30 13:15:20.917251 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:20.917224 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a3e01-29889435-tqtlv"] Oct 30 13:15:21.294412 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.294378 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889435-h2hfn"] Oct 30 13:15:21.299497 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.299466 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889435-q4q4t"] Oct 30 13:15:21.355625 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.355597 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2" Oct 30 13:15:21.356104 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.355606 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2" event={"ID":"2a5f8729-df67-4e02-b7c5-f2250efa9678","Type":"ContainerDied","Data":"23900db97ceb9dc12b5c70cb68c4508dc57fd163d5aae5a50a089d8025494f7d"} Oct 30 13:15:21.356104 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.355723 4931 scope.go:122] "RemoveContainer" containerID="d25a4623fc82f949dd783fe31a56a31af9957f2c5ad87524f6d79a6e7b08ae78" Oct 30 13:15:21.359364 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.359343 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-14e8a4-29889435-cldf2" Oct 30 13:15:21.359364 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.359345 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-14e8a4-29889435-cldf2" event={"ID":"d882efb1-cd4f-45ce-b829-4795003e31b9","Type":"ContainerDied","Data":"43bb5c6868b99cbc1e7305441d4e03d39028879a995f7094f638aa715164a756"} Oct 30 13:15:21.369215 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.369108 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4a8ee1-29889435-5w88z" event={"ID":"ec2e9458-537c-428a-abbe-6ff129af3c6e","Type":"ContainerDied","Data":"a90b35c7329f1ea86afd646e100e7a08ff7ce2cbc4f6ca0bc62170679da272b5"} Oct 30 13:15:21.369215 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.369083 4931 generic.go:356] "Generic (PLEG): container finished" podID="ec2e9458-537c-428a-abbe-6ff129af3c6e" containerID="a90b35c7329f1ea86afd646e100e7a08ff7ce2cbc4f6ca0bc62170679da272b5" exitCode=60 Oct 30 13:15:21.369880 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.369572 4931 scope.go:122] "RemoveContainer" containerID="a0b5b2e700f9e85b64f45f656b57b5f4e952c92a6ad02314b821d0aa9f07785e" Oct 30 13:15:21.387964 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.383709 4931 generic.go:356] "Generic (PLEG): container finished" podID="795f8222-5020-4ef8-be21-49fbb8b178e8" containerID="d3ed2be84e41e8df396d109efb2dd2a4653de759cdefa5602c9b2497add05b0e" exitCode=60 Oct 30 13:15:21.387964 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.383803 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889435-q4q4t" event={"ID":"795f8222-5020-4ef8-be21-49fbb8b178e8","Type":"ContainerDied","Data":"d3ed2be84e41e8df396d109efb2dd2a4653de759cdefa5602c9b2497add05b0e"} Oct 30 13:15:21.399817 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.399548 4931 generic.go:356] "Generic (PLEG): container finished" podID="f06e3aa4-5df5-45c3-a80e-02b922c62768" containerID="f87a0ef7626b712567a9e9e380f386322a9b9425bf9ce5a37ed9306dca6bb7fd" exitCode=60 Oct 30 13:15:21.399817 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.399591 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-h2hfn" event={"ID":"f06e3aa4-5df5-45c3-a80e-02b922c62768","Type":"ContainerDied","Data":"f87a0ef7626b712567a9e9e380f386322a9b9425bf9ce5a37ed9306dca6bb7fd"} Oct 30 13:15:21.404874 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.404847 4931 scope.go:122] "RemoveContainer" containerID="8679f81ed0b750b1b32ca024eb9600ee302b7051fd6c3ce9c5512cde4b99544b" Oct 30 13:15:21.427113 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.427083 4931 scope.go:122] "RemoveContainer" containerID="81a563c30bab377fb31f64628df2799df51e6729a7db27da9700da0525c1f6ea" Oct 30 13:15:21.432773 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.431889 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2"] Oct 30 13:15:21.439338 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.437742 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889435-fm9c2"] Oct 30 13:15:21.461425 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.457952 4931 scope.go:122] "RemoveContainer" containerID="34ba8f6af56ac562bbd3dae00f48a6cafdfe60fadb1b551523729c93d136108f" Oct 30 13:15:21.493306 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.492151 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-14e8a4-29889435-cldf2"] Oct 30 13:15:21.506317 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.504244 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-14e8a4-29889435-cldf2"] Oct 30 13:15:21.523491 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.522680 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08e53b66-2198-4d37-82d5-563f36393d67" path="/var/lib/kubelet/pods/08e53b66-2198-4d37-82d5-563f36393d67/volumes" Oct 30 13:15:21.523676 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.523557 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a5f8729-df67-4e02-b7c5-f2250efa9678" path="/var/lib/kubelet/pods/2a5f8729-df67-4e02-b7c5-f2250efa9678/volumes" Oct 30 13:15:21.524753 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.524261 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d882efb1-cd4f-45ce-b829-4795003e31b9" path="/var/lib/kubelet/pods/d882efb1-cd4f-45ce-b829-4795003e31b9/volumes" Oct 30 13:15:21.631733 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.631703 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj" Oct 30 13:15:21.675327 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.674907 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3bc7915-8ed0-47c5-a693-15db3d153fe8-kube-api-access-tpxxf\" (UniqueName: \"kubernetes.io/projected/c3bc7915-8ed0-47c5-a693-15db3d153fe8-kube-api-access-tpxxf\") pod \"c3bc7915-8ed0-47c5-a693-15db3d153fe8\" (UID: \"c3bc7915-8ed0-47c5-a693-15db3d153fe8\") " Oct 30 13:15:21.675327 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.674988 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3bc7915-8ed0-47c5-a693-15db3d153fe8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3bc7915-8ed0-47c5-a693-15db3d153fe8-maas-api-service-ca\") pod \"c3bc7915-8ed0-47c5-a693-15db3d153fe8\" (UID: \"c3bc7915-8ed0-47c5-a693-15db3d153fe8\") " Oct 30 13:15:21.680526 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.679917 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3bc7915-8ed0-47c5-a693-15db3d153fe8-maas-api-service-ca" pod "c3bc7915-8ed0-47c5-a693-15db3d153fe8" (UID: "c3bc7915-8ed0-47c5-a693-15db3d153fe8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:15:21.687109 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.687065 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3bc7915-8ed0-47c5-a693-15db3d153fe8-kube-api-access-tpxxf" pod "c3bc7915-8ed0-47c5-a693-15db3d153fe8" (UID: "c3bc7915-8ed0-47c5-a693-15db3d153fe8"). InnerVolumeSpecName "kube-api-access-tpxxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:21.714927 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.714441 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4a8ee1-29889435-5w88z" Oct 30 13:15:21.749601 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.748764 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-h2hfn" Oct 30 13:15:21.780913 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.780883 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec2e9458-537c-428a-abbe-6ff129af3c6e-kube-api-access-xw464\" (UniqueName: \"kubernetes.io/projected/ec2e9458-537c-428a-abbe-6ff129af3c6e-kube-api-access-xw464\") pod \"ec2e9458-537c-428a-abbe-6ff129af3c6e\" (UID: \"ec2e9458-537c-428a-abbe-6ff129af3c6e\") " Oct 30 13:15:21.781099 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.781080 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f06e3aa4-5df5-45c3-a80e-02b922c62768-kube-api-access-ffmq8\" (UniqueName: \"kubernetes.io/projected/f06e3aa4-5df5-45c3-a80e-02b922c62768-kube-api-access-ffmq8\") pod \"f06e3aa4-5df5-45c3-a80e-02b922c62768\" (UID: \"f06e3aa4-5df5-45c3-a80e-02b922c62768\") " Oct 30 13:15:21.781229 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.781201 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec2e9458-537c-428a-abbe-6ff129af3c6e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2e9458-537c-428a-abbe-6ff129af3c6e-maas-api-service-ca\") pod \"ec2e9458-537c-428a-abbe-6ff129af3c6e\" (UID: \"ec2e9458-537c-428a-abbe-6ff129af3c6e\") " Oct 30 13:15:21.781396 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.781252 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f06e3aa4-5df5-45c3-a80e-02b922c62768-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f06e3aa4-5df5-45c3-a80e-02b922c62768-maas-api-service-ca\") pod \"f06e3aa4-5df5-45c3-a80e-02b922c62768\" (UID: \"f06e3aa4-5df5-45c3-a80e-02b922c62768\") " Oct 30 13:15:21.781697 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.781664 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec2e9458-537c-428a-abbe-6ff129af3c6e-maas-api-service-ca" pod "ec2e9458-537c-428a-abbe-6ff129af3c6e" (UID: "ec2e9458-537c-428a-abbe-6ff129af3c6e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:15:21.781913 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.781856 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpxxf\" (UniqueName: \"kubernetes.io/projected/c3bc7915-8ed0-47c5-a693-15db3d153fe8-kube-api-access-tpxxf\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:15:21.781913 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.781877 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3bc7915-8ed0-47c5-a693-15db3d153fe8-maas-api-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:15:21.781913 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.781887 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2e9458-537c-428a-abbe-6ff129af3c6e-maas-api-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:15:21.782069 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.782003 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f06e3aa4-5df5-45c3-a80e-02b922c62768-maas-api-service-ca" pod "f06e3aa4-5df5-45c3-a80e-02b922c62768" (UID: "f06e3aa4-5df5-45c3-a80e-02b922c62768"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:15:21.783777 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.783750 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec2e9458-537c-428a-abbe-6ff129af3c6e-kube-api-access-xw464" pod "ec2e9458-537c-428a-abbe-6ff129af3c6e" (UID: "ec2e9458-537c-428a-abbe-6ff129af3c6e"). InnerVolumeSpecName "kube-api-access-xw464". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:21.785542 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.785513 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f06e3aa4-5df5-45c3-a80e-02b922c62768-kube-api-access-ffmq8" pod "f06e3aa4-5df5-45c3-a80e-02b922c62768" (UID: "f06e3aa4-5df5-45c3-a80e-02b922c62768"). InnerVolumeSpecName "kube-api-access-ffmq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:21.826685 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.826617 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889435-q4q4t" Oct 30 13:15:21.834024 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.833998 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh" Oct 30 13:15:21.883411 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.883369 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a78c34-1e58-48f3-8660-56d6b19d8847-kube-api-access-9btl9\" (UniqueName: \"kubernetes.io/projected/01a78c34-1e58-48f3-8660-56d6b19d8847-kube-api-access-9btl9\") pod \"01a78c34-1e58-48f3-8660-56d6b19d8847\" (UID: \"01a78c34-1e58-48f3-8660-56d6b19d8847\") " Oct 30 13:15:21.883612 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.883484 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/795f8222-5020-4ef8-be21-49fbb8b178e8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/795f8222-5020-4ef8-be21-49fbb8b178e8-maas-api-service-ca\") pod \"795f8222-5020-4ef8-be21-49fbb8b178e8\" (UID: \"795f8222-5020-4ef8-be21-49fbb8b178e8\") " Oct 30 13:15:21.883612 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.883597 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/795f8222-5020-4ef8-be21-49fbb8b178e8-kube-api-access-ktxdj\" (UniqueName: \"kubernetes.io/projected/795f8222-5020-4ef8-be21-49fbb8b178e8-kube-api-access-ktxdj\") pod \"795f8222-5020-4ef8-be21-49fbb8b178e8\" (UID: \"795f8222-5020-4ef8-be21-49fbb8b178e8\") " Oct 30 13:15:21.883796 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.883664 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01a78c34-1e58-48f3-8660-56d6b19d8847-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a78c34-1e58-48f3-8660-56d6b19d8847-maas-api-service-ca\") pod \"01a78c34-1e58-48f3-8660-56d6b19d8847\" (UID: \"01a78c34-1e58-48f3-8660-56d6b19d8847\") " Oct 30 13:15:21.884467 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.884431 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a78c34-1e58-48f3-8660-56d6b19d8847-maas-api-service-ca" pod "01a78c34-1e58-48f3-8660-56d6b19d8847" (UID: "01a78c34-1e58-48f3-8660-56d6b19d8847"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:15:21.884698 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.884577 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f06e3aa4-5df5-45c3-a80e-02b922c62768-maas-api-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:15:21.884698 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.884628 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw464\" (UniqueName: \"kubernetes.io/projected/ec2e9458-537c-428a-abbe-6ff129af3c6e-kube-api-access-xw464\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:15:21.884698 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.884648 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffmq8\" (UniqueName: \"kubernetes.io/projected/f06e3aa4-5df5-45c3-a80e-02b922c62768-kube-api-access-ffmq8\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:15:21.885182 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.885157 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/795f8222-5020-4ef8-be21-49fbb8b178e8-maas-api-service-ca" pod "795f8222-5020-4ef8-be21-49fbb8b178e8" (UID: "795f8222-5020-4ef8-be21-49fbb8b178e8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:15:21.888093 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.887926 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a78c34-1e58-48f3-8660-56d6b19d8847-kube-api-access-9btl9" pod "01a78c34-1e58-48f3-8660-56d6b19d8847" (UID: "01a78c34-1e58-48f3-8660-56d6b19d8847"). InnerVolumeSpecName "kube-api-access-9btl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:21.888547 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.888511 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/795f8222-5020-4ef8-be21-49fbb8b178e8-kube-api-access-ktxdj" pod "795f8222-5020-4ef8-be21-49fbb8b178e8" (UID: "795f8222-5020-4ef8-be21-49fbb8b178e8"). InnerVolumeSpecName "kube-api-access-ktxdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:21.986361 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.986322 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a78c34-1e58-48f3-8660-56d6b19d8847-maas-api-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:15:21.986361 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.986362 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9btl9\" (UniqueName: \"kubernetes.io/projected/01a78c34-1e58-48f3-8660-56d6b19d8847-kube-api-access-9btl9\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:15:21.986578 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.986377 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/795f8222-5020-4ef8-be21-49fbb8b178e8-maas-api-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:15:21.986578 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:21.986390 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktxdj\" (UniqueName: \"kubernetes.io/projected/795f8222-5020-4ef8-be21-49fbb8b178e8-kube-api-access-ktxdj\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:15:22.410960 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:22.410925 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj" Oct 30 13:15:22.410960 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:22.410942 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj" event={"ID":"c3bc7915-8ed0-47c5-a693-15db3d153fe8","Type":"ContainerDied","Data":"e21c3079756b17c59380f55db9f63ae31853bbcdedaa5ccffcc3b79bfa9a948b"} Oct 30 13:15:22.411547 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:22.411005 4931 scope.go:122] "RemoveContainer" containerID="5b9ef4093290232d4ebf4ec3cde5f426dcbfd6d8fbf67feb7251fd809b5c15a3" Oct 30 13:15:22.413661 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:22.413640 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4a8ee1-29889435-5w88z" Oct 30 13:15:22.413778 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:22.413683 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4a8ee1-29889435-5w88z" event={"ID":"ec2e9458-537c-428a-abbe-6ff129af3c6e","Type":"ContainerDied","Data":"f7fa618e7d6e1b6ceff3a0f8301032fea00d82d34ae6846c88d0f7486bca31d3"} Oct 30 13:15:22.415255 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:22.415230 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889435-q4q4t" Oct 30 13:15:22.415376 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:22.415278 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889435-q4q4t" event={"ID":"795f8222-5020-4ef8-be21-49fbb8b178e8","Type":"ContainerDied","Data":"2dd3c2cd8bebce9ae76a506d53730e16a15b7476a960f8d0006db2c3e5c2296c"} Oct 30 13:15:22.417305 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:22.417254 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh" Oct 30 13:15:22.417408 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:22.417303 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh" event={"ID":"01a78c34-1e58-48f3-8660-56d6b19d8847","Type":"ContainerDied","Data":"20b3cb4786b13a323e7db741aaeedff259324c672c25c1b9092e03a677f84e3f"} Oct 30 13:15:22.422609 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:22.422450 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-h2hfn" event={"ID":"f06e3aa4-5df5-45c3-a80e-02b922c62768","Type":"ContainerDied","Data":"ec0d9de09e50436fb27fdca1be194030530b372864a36e1d419bb24fbcb2c62f"} Oct 30 13:15:22.422609 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:22.422565 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-h2hfn" Oct 30 13:15:22.425442 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:22.425384 4931 scope.go:122] "RemoveContainer" containerID="a90b35c7329f1ea86afd646e100e7a08ff7ce2cbc4f6ca0bc62170679da272b5" Oct 30 13:15:22.441161 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:22.441140 4931 scope.go:122] "RemoveContainer" containerID="d3ed2be84e41e8df396d109efb2dd2a4653de759cdefa5602c9b2497add05b0e" Oct 30 13:15:22.451100 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:22.451081 4931 scope.go:122] "RemoveContainer" containerID="46d2fd777f87a2996dc63d919032f2f639294d7fe2b69eb497a6ddf6e42184ad" Oct 30 13:15:22.461143 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:22.461123 4931 scope.go:122] "RemoveContainer" containerID="f87a0ef7626b712567a9e9e380f386322a9b9425bf9ce5a37ed9306dca6bb7fd" Oct 30 13:15:22.519926 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:22.519893 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889435-q4q4t"] Oct 30 13:15:22.531380 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:22.531350 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889435-q4q4t"] Oct 30 13:15:22.577407 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:22.577366 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj"] Oct 30 13:15:22.593945 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:22.593915 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889435-ngfbj"] Oct 30 13:15:22.633452 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:22.633413 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4a8ee1-29889435-5w88z"] Oct 30 13:15:22.638903 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:22.638877 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4a8ee1-29889435-5w88z"] Oct 30 13:15:22.692087 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:22.692026 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh"] Oct 30 13:15:22.699642 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:22.699614 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889435-gktgh"] Oct 30 13:15:22.742609 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:22.742575 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889435-h2hfn"] Oct 30 13:15:22.747819 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:22.747793 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889435-h2hfn"] Oct 30 13:15:23.514655 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:23.514620 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01a78c34-1e58-48f3-8660-56d6b19d8847" path="/var/lib/kubelet/pods/01a78c34-1e58-48f3-8660-56d6b19d8847/volumes" Oct 30 13:15:23.515336 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:23.515311 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="795f8222-5020-4ef8-be21-49fbb8b178e8" path="/var/lib/kubelet/pods/795f8222-5020-4ef8-be21-49fbb8b178e8/volumes" Oct 30 13:15:23.515880 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:23.515831 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3bc7915-8ed0-47c5-a693-15db3d153fe8" path="/var/lib/kubelet/pods/c3bc7915-8ed0-47c5-a693-15db3d153fe8/volumes" Oct 30 13:15:23.516756 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:23.516737 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec2e9458-537c-428a-abbe-6ff129af3c6e" path="/var/lib/kubelet/pods/ec2e9458-537c-428a-abbe-6ff129af3c6e/volumes" Oct 30 13:15:23.517281 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:23.517252 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f06e3aa4-5df5-45c3-a80e-02b922c62768" path="/var/lib/kubelet/pods/f06e3aa4-5df5-45c3-a80e-02b922c62768/volumes" Oct 30 13:15:27.063470 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:27.063433 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4189e66-j5hfv"] Oct 30 13:15:27.073409 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:27.073381 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4189e66-j5hfv"] Oct 30 13:15:27.514565 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:27.514518 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66dae659-dcb6-4bd9-97f6-8ec85c68dc8e" path="/var/lib/kubelet/pods/66dae659-dcb6-4bd9-97f6-8ec85c68dc8e/volumes" Oct 30 13:15:34.206671 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:34.206626 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7"] Oct 30 13:15:34.207156 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:34.206891 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7" podUID="6acaa883-abd5-40b0-b62d-2aaf38ad32ea" containerName="maas-api" containerID="cri-o://6b11d1d7e7819dc83615ab3b1017e3eace3ab7c64569cf5919e1278c388fa2b7" gracePeriod=30 Oct 30 13:15:34.700507 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:34.681798 4931 generic.go:356] "Generic (PLEG): container finished" podID="6acaa883-abd5-40b0-b62d-2aaf38ad32ea" containerID="6b11d1d7e7819dc83615ab3b1017e3eace3ab7c64569cf5919e1278c388fa2b7" exitCode=0 Oct 30 13:15:34.700507 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:34.681855 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7" event={"ID":"6acaa883-abd5-40b0-b62d-2aaf38ad32ea","Type":"ContainerDied","Data":"6b11d1d7e7819dc83615ab3b1017e3eace3ab7c64569cf5919e1278c388fa2b7"} Oct 30 13:15:34.733112 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:34.730575 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7" Oct 30 13:15:34.771038 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:34.771002 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6acaa883-abd5-40b0-b62d-2aaf38ad32ea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6acaa883-abd5-40b0-b62d-2aaf38ad32ea-metrics-certs\") pod \"6acaa883-abd5-40b0-b62d-2aaf38ad32ea\" (UID: \"6acaa883-abd5-40b0-b62d-2aaf38ad32ea\") " Oct 30 13:15:34.771217 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:34.771120 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6acaa883-abd5-40b0-b62d-2aaf38ad32ea-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6acaa883-abd5-40b0-b62d-2aaf38ad32ea-openshift-service-ca\") pod \"6acaa883-abd5-40b0-b62d-2aaf38ad32ea\" (UID: \"6acaa883-abd5-40b0-b62d-2aaf38ad32ea\") " Oct 30 13:15:34.771346 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:34.771294 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6acaa883-abd5-40b0-b62d-2aaf38ad32ea-kube-api-access-vgp6x\" (UniqueName: \"kubernetes.io/projected/6acaa883-abd5-40b0-b62d-2aaf38ad32ea-kube-api-access-vgp6x\") pod \"6acaa883-abd5-40b0-b62d-2aaf38ad32ea\" (UID: \"6acaa883-abd5-40b0-b62d-2aaf38ad32ea\") " Oct 30 13:15:34.771408 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:34.771356 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6acaa883-abd5-40b0-b62d-2aaf38ad32ea-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6acaa883-abd5-40b0-b62d-2aaf38ad32ea-maas-api-tls\") pod \"6acaa883-abd5-40b0-b62d-2aaf38ad32ea\" (UID: \"6acaa883-abd5-40b0-b62d-2aaf38ad32ea\") " Oct 30 13:15:34.771643 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:34.771613 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6acaa883-abd5-40b0-b62d-2aaf38ad32ea-openshift-service-ca" pod "6acaa883-abd5-40b0-b62d-2aaf38ad32ea" (UID: "6acaa883-abd5-40b0-b62d-2aaf38ad32ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:15:34.772037 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:34.772012 4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6acaa883-abd5-40b0-b62d-2aaf38ad32ea-openshift-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:15:34.774359 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:34.774328 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6acaa883-abd5-40b0-b62d-2aaf38ad32ea-metrics-certs" pod "6acaa883-abd5-40b0-b62d-2aaf38ad32ea" (UID: "6acaa883-abd5-40b0-b62d-2aaf38ad32ea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:15:34.776654 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:34.776627 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6acaa883-abd5-40b0-b62d-2aaf38ad32ea-maas-api-tls" pod "6acaa883-abd5-40b0-b62d-2aaf38ad32ea" (UID: "6acaa883-abd5-40b0-b62d-2aaf38ad32ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:15:34.779817 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:34.779787 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6acaa883-abd5-40b0-b62d-2aaf38ad32ea-kube-api-access-vgp6x" pod "6acaa883-abd5-40b0-b62d-2aaf38ad32ea" (UID: "6acaa883-abd5-40b0-b62d-2aaf38ad32ea"). InnerVolumeSpecName "kube-api-access-vgp6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:34.873093 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:34.873058 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgp6x\" (UniqueName: \"kubernetes.io/projected/6acaa883-abd5-40b0-b62d-2aaf38ad32ea-kube-api-access-vgp6x\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:15:34.873093 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:34.873090 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6acaa883-abd5-40b0-b62d-2aaf38ad32ea-maas-api-tls\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:15:34.873093 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:34.873100 4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6acaa883-abd5-40b0-b62d-2aaf38ad32ea-metrics-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:15:35.137284 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:35.137233 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31e10b-75k7t"] Oct 30 13:15:35.141209 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:35.141184 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31e10b-75k7t"] Oct 30 13:15:35.514054 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:35.514024 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3037fde1-6ac2-4d15-8ecc-354dee56569c" path="/var/lib/kubelet/pods/3037fde1-6ac2-4d15-8ecc-354dee56569c/volumes" Oct 30 13:15:35.691329 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:35.691296 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7" Oct 30 13:15:35.691329 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:35.691316 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7" event={"ID":"6acaa883-abd5-40b0-b62d-2aaf38ad32ea","Type":"ContainerDied","Data":"cd912d75dddc1d132dc5b93a749b508ac0dc2bc87b59001a5681897be1870561"} Oct 30 13:15:35.691550 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:35.691404 4931 scope.go:122] "RemoveContainer" containerID="6b11d1d7e7819dc83615ab3b1017e3eace3ab7c64569cf5919e1278c388fa2b7" Oct 30 13:15:35.750570 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:35.750535 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7"] Oct 30 13:15:35.764180 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:35.764116 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-31e10b-6b6b9665cf-lkmt7"] Oct 30 13:15:37.353451 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:15:37.353416 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:15:37.515126 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:37.515092 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6acaa883-abd5-40b0-b62d-2aaf38ad32ea" path="/var/lib/kubelet/pods/6acaa883-abd5-40b0-b62d-2aaf38ad32ea/volumes" Oct 30 13:15:43.790346 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:43.778644 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:15:46.055008 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:46.054969 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4ec3f23-h7q9k"] Oct 30 13:15:46.065076 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:46.065047 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4ec3f23-h7q9k"] Oct 30 13:15:47.515283 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:47.515222 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="519b9db0-c559-4f08-be61-ea559d8a1189" path="/var/lib/kubelet/pods/519b9db0-c559-4f08-be61-ea559d8a1189/volumes" Oct 30 13:15:57.711868 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.711833 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d"] Oct 30 13:15:57.712442 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712427 4931 cpu_manager.go:414] "containerMap: removing container" podUID="f06e3aa4-5df5-45c3-a80e-02b922c62768" containerName="cleanup" Oct 30 13:15:57.712497 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712443 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f06e3aa4-5df5-45c3-a80e-02b922c62768" containerName="cleanup" Oct 30 13:15:57.712497 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712455 4931 cpu_manager.go:414] "containerMap: removing container" podUID="d882efb1-cd4f-45ce-b829-4795003e31b9" containerName="cleanup" Oct 30 13:15:57.712497 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712461 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d882efb1-cd4f-45ce-b829-4795003e31b9" containerName="cleanup" Oct 30 13:15:57.712497 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712468 4931 cpu_manager.go:414] "containerMap: removing container" podUID="f06e3aa4-5df5-45c3-a80e-02b922c62768" containerName="cleanup" Oct 30 13:15:57.712497 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712472 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f06e3aa4-5df5-45c3-a80e-02b922c62768" containerName="cleanup" Oct 30 13:15:57.712497 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712482 4931 cpu_manager.go:414] "containerMap: removing container" podUID="01a78c34-1e58-48f3-8660-56d6b19d8847" containerName="cleanup" Oct 30 13:15:57.712497 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712487 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01a78c34-1e58-48f3-8660-56d6b19d8847" containerName="cleanup" Oct 30 13:15:57.712497 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712496 4931 cpu_manager.go:414] "containerMap: removing container" podUID="6acaa883-abd5-40b0-b62d-2aaf38ad32ea" containerName="maas-api" Oct 30 13:15:57.712497 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712501 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6acaa883-abd5-40b0-b62d-2aaf38ad32ea" containerName="maas-api" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712507 4931 cpu_manager.go:414] "containerMap: removing container" podUID="c3bc7915-8ed0-47c5-a693-15db3d153fe8" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712512 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3bc7915-8ed0-47c5-a693-15db3d153fe8" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712519 4931 cpu_manager.go:414] "containerMap: removing container" podUID="2a5f8729-df67-4e02-b7c5-f2250efa9678" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712524 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a5f8729-df67-4e02-b7c5-f2250efa9678" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712530 4931 cpu_manager.go:414] "containerMap: removing container" podUID="ea3c064a-c226-4eec-bd7e-d42fb03965ad" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712535 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea3c064a-c226-4eec-bd7e-d42fb03965ad" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712548 4931 cpu_manager.go:414] "containerMap: removing container" podUID="d882efb1-cd4f-45ce-b829-4795003e31b9" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712553 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d882efb1-cd4f-45ce-b829-4795003e31b9" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712561 4931 cpu_manager.go:414] "containerMap: removing container" podUID="2a5f8729-df67-4e02-b7c5-f2250efa9678" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712565 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a5f8729-df67-4e02-b7c5-f2250efa9678" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712576 4931 cpu_manager.go:414] "containerMap: removing container" podUID="29a14a0f-98aa-4981-abe4-56919051950b" containerName="istio-proxy" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712580 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29a14a0f-98aa-4981-abe4-56919051950b" containerName="istio-proxy" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712586 4931 cpu_manager.go:414] "containerMap: removing container" podUID="ec2e9458-537c-428a-abbe-6ff129af3c6e" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712591 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec2e9458-537c-428a-abbe-6ff129af3c6e" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712598 4931 cpu_manager.go:414] "containerMap: removing container" podUID="08e53b66-2198-4d37-82d5-563f36393d67" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712602 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08e53b66-2198-4d37-82d5-563f36393d67" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712612 4931 cpu_manager.go:414] "containerMap: removing container" podUID="ec2e9458-537c-428a-abbe-6ff129af3c6e" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712616 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec2e9458-537c-428a-abbe-6ff129af3c6e" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712624 4931 cpu_manager.go:414] "containerMap: removing container" podUID="795f8222-5020-4ef8-be21-49fbb8b178e8" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712628 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="795f8222-5020-4ef8-be21-49fbb8b178e8" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712636 4931 cpu_manager.go:414] "containerMap: removing container" podUID="0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712640 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712645 4931 cpu_manager.go:414] "containerMap: removing container" podUID="f1814684-56ff-4b44-8328-29cbe0497bf2" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712649 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1814684-56ff-4b44-8328-29cbe0497bf2" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712655 4931 cpu_manager.go:414] "containerMap: removing container" podUID="ea3c064a-c226-4eec-bd7e-d42fb03965ad" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712660 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea3c064a-c226-4eec-bd7e-d42fb03965ad" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712666 4931 cpu_manager.go:414] "containerMap: removing container" podUID="0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712670 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712678 4931 cpu_manager.go:414] "containerMap: removing container" podUID="d882efb1-cd4f-45ce-b829-4795003e31b9" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712683 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d882efb1-cd4f-45ce-b829-4795003e31b9" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712688 4931 cpu_manager.go:414] "containerMap: removing container" podUID="01a78c34-1e58-48f3-8660-56d6b19d8847" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712692 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01a78c34-1e58-48f3-8660-56d6b19d8847" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712700 4931 cpu_manager.go:414] "containerMap: removing container" podUID="0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712705 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712712 4931 cpu_manager.go:414] "containerMap: removing container" podUID="ec2e9458-537c-428a-abbe-6ff129af3c6e" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712716 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec2e9458-537c-428a-abbe-6ff129af3c6e" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712723 4931 cpu_manager.go:414] "containerMap: removing container" podUID="795f8222-5020-4ef8-be21-49fbb8b178e8" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712727 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="795f8222-5020-4ef8-be21-49fbb8b178e8" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712736 4931 cpu_manager.go:414] "containerMap: removing container" podUID="08e53b66-2198-4d37-82d5-563f36393d67" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712741 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08e53b66-2198-4d37-82d5-563f36393d67" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712747 4931 cpu_manager.go:414] "containerMap: removing container" podUID="c3bc7915-8ed0-47c5-a693-15db3d153fe8" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712752 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3bc7915-8ed0-47c5-a693-15db3d153fe8" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712768 4931 cpu_manager.go:414] "containerMap: removing container" podUID="f1814684-56ff-4b44-8328-29cbe0497bf2" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712773 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1814684-56ff-4b44-8328-29cbe0497bf2" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712778 4931 cpu_manager.go:414] "containerMap: removing container" podUID="f1814684-56ff-4b44-8328-29cbe0497bf2" containerName="cleanup" Oct 30 13:15:57.712823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712782 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1814684-56ff-4b44-8328-29cbe0497bf2" containerName="cleanup" Oct 30 13:15:57.714170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712882 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="08e53b66-2198-4d37-82d5-563f36393d67" containerName="cleanup" Oct 30 13:15:57.714170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712889 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3bc7915-8ed0-47c5-a693-15db3d153fe8" containerName="cleanup" Oct 30 13:15:57.714170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712894 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f06e3aa4-5df5-45c3-a80e-02b922c62768" containerName="cleanup" Oct 30 13:15:57.714170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712900 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea3c064a-c226-4eec-bd7e-d42fb03965ad" containerName="cleanup" Oct 30 13:15:57.714170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712907 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="08e53b66-2198-4d37-82d5-563f36393d67" containerName="cleanup" Oct 30 13:15:57.714170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712916 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1814684-56ff-4b44-8328-29cbe0497bf2" containerName="cleanup" Oct 30 13:15:57.714170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712921 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1814684-56ff-4b44-8328-29cbe0497bf2" containerName="cleanup" Oct 30 13:15:57.714170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712927 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="d882efb1-cd4f-45ce-b829-4795003e31b9" containerName="cleanup" Oct 30 13:15:57.714170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712932 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f06e3aa4-5df5-45c3-a80e-02b922c62768" containerName="cleanup" Oct 30 13:15:57.714170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712938 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="01a78c34-1e58-48f3-8660-56d6b19d8847" containerName="cleanup" Oct 30 13:15:57.714170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712943 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6acaa883-abd5-40b0-b62d-2aaf38ad32ea" containerName="maas-api" Oct 30 13:15:57.714170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712950 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3bc7915-8ed0-47c5-a693-15db3d153fe8" containerName="cleanup" Oct 30 13:15:57.714170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712955 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a5f8729-df67-4e02-b7c5-f2250efa9678" containerName="cleanup" Oct 30 13:15:57.714170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712959 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea3c064a-c226-4eec-bd7e-d42fb03965ad" containerName="cleanup" Oct 30 13:15:57.714170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712966 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="d882efb1-cd4f-45ce-b829-4795003e31b9" containerName="cleanup" Oct 30 13:15:57.714170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712971 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a5f8729-df67-4e02-b7c5-f2250efa9678" containerName="cleanup" Oct 30 13:15:57.714170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712976 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="01a78c34-1e58-48f3-8660-56d6b19d8847" containerName="cleanup" Oct 30 13:15:57.714170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712981 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="29a14a0f-98aa-4981-abe4-56919051950b" containerName="istio-proxy" Oct 30 13:15:57.714170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712986 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec2e9458-537c-428a-abbe-6ff129af3c6e" containerName="cleanup" Oct 30 13:15:57.714170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712991 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="795f8222-5020-4ef8-be21-49fbb8b178e8" containerName="cleanup" Oct 30 13:15:57.714170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.712997 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b" containerName="cleanup" Oct 30 13:15:57.714170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.713002 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1814684-56ff-4b44-8328-29cbe0497bf2" containerName="cleanup" Oct 30 13:15:57.714170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.713007 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b" containerName="cleanup" Oct 30 13:15:57.714170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.713013 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="d882efb1-cd4f-45ce-b829-4795003e31b9" containerName="cleanup" Oct 30 13:15:57.714170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.713017 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="01a78c34-1e58-48f3-8660-56d6b19d8847" containerName="cleanup" Oct 30 13:15:57.714170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.713023 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a5f8729-df67-4e02-b7c5-f2250efa9678" containerName="cleanup" Oct 30 13:15:57.714170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.713028 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="0597dc4a-f61d-4eff-a9fe-5ec26a9c8e4b" containerName="cleanup" Oct 30 13:15:57.714170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.713034 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec2e9458-537c-428a-abbe-6ff129af3c6e" containerName="cleanup" Oct 30 13:15:57.714170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.713040 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="795f8222-5020-4ef8-be21-49fbb8b178e8" containerName="cleanup" Oct 30 13:15:57.714170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.713047 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="795f8222-5020-4ef8-be21-49fbb8b178e8" containerName="cleanup" Oct 30 13:15:57.716708 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.716690 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d" Oct 30 13:15:57.742338 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.742313 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d"] Oct 30 13:15:57.751020 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.750994 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-6af611-serving-cert\"" Oct 30 13:15:57.887768 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.887735 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a9754b7-dc87-483f-9164-037a383d1f99-metrics-certs\") pod \"maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d\" (UID: \"1a9754b7-dc87-483f-9164-037a383d1f99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d" Oct 30 13:15:57.887925 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.887800 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rpjl\" (UniqueName: \"kubernetes.io/projected/1a9754b7-dc87-483f-9164-037a383d1f99-kube-api-access-5rpjl\") pod \"maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d\" (UID: \"1a9754b7-dc87-483f-9164-037a383d1f99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d" Oct 30 13:15:57.888017 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.887999 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a9754b7-dc87-483f-9164-037a383d1f99-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d\" (UID: \"1a9754b7-dc87-483f-9164-037a383d1f99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d" Oct 30 13:15:57.888172 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.888156 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a9754b7-dc87-483f-9164-037a383d1f99-maas-api-tls\") pod \"maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d\" (UID: \"1a9754b7-dc87-483f-9164-037a383d1f99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d" Oct 30 13:15:57.989907 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.989804 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a9754b7-dc87-483f-9164-037a383d1f99-metrics-certs\") pod \"maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d\" (UID: \"1a9754b7-dc87-483f-9164-037a383d1f99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d" Oct 30 13:15:57.989907 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.989890 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rpjl\" (UniqueName: \"kubernetes.io/projected/1a9754b7-dc87-483f-9164-037a383d1f99-kube-api-access-5rpjl\") pod \"maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d\" (UID: \"1a9754b7-dc87-483f-9164-037a383d1f99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d" Oct 30 13:15:57.990155 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.990023 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a9754b7-dc87-483f-9164-037a383d1f99-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d\" (UID: \"1a9754b7-dc87-483f-9164-037a383d1f99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d" Oct 30 13:15:57.990214 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.990164 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a9754b7-dc87-483f-9164-037a383d1f99-maas-api-tls\") pod \"maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d\" (UID: \"1a9754b7-dc87-483f-9164-037a383d1f99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d" Oct 30 13:15:57.990823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.990794 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a9754b7-dc87-483f-9164-037a383d1f99-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d\" (UID: \"1a9754b7-dc87-483f-9164-037a383d1f99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d" Oct 30 13:15:57.992924 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.992896 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a9754b7-dc87-483f-9164-037a383d1f99-maas-api-tls\") pod \"maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d\" (UID: \"1a9754b7-dc87-483f-9164-037a383d1f99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d" Oct 30 13:15:57.993147 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:57.992907 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a9754b7-dc87-483f-9164-037a383d1f99-metrics-certs\") pod \"maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d\" (UID: \"1a9754b7-dc87-483f-9164-037a383d1f99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d" Oct 30 13:15:58.015175 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:58.015138 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rpjl\" (UniqueName: \"kubernetes.io/projected/1a9754b7-dc87-483f-9164-037a383d1f99-kube-api-access-5rpjl\") pod \"maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d\" (UID: \"1a9754b7-dc87-483f-9164-037a383d1f99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d" Oct 30 13:15:58.030068 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:58.030033 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d" Oct 30 13:15:58.357618 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:15:58.357580 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a9754b7_dc87_483f_9164_037a383d1f99.slice/crio-bac1be8dd6920e011970aa0e3a39ea8ae2c76b6ea771699e3cb4a9034e2be4aa WatchSource:0}: Error finding container bac1be8dd6920e011970aa0e3a39ea8ae2c76b6ea771699e3cb4a9034e2be4aa: Status 404 returned error can't find the container with id bac1be8dd6920e011970aa0e3a39ea8ae2c76b6ea771699e3cb4a9034e2be4aa Oct 30 13:15:58.357754 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:58.357612 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d"] Oct 30 13:15:58.988787 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:58.988745 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d" event={"ID":"1a9754b7-dc87-483f-9164-037a383d1f99","Type":"ContainerStarted","Data":"0b663915cd9de167730ed79227bb1a23ce289e127d088d51042b7ae6df025d7e"} Oct 30 13:15:58.989193 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:58.988809 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d" event={"ID":"1a9754b7-dc87-483f-9164-037a383d1f99","Type":"ContainerStarted","Data":"bac1be8dd6920e011970aa0e3a39ea8ae2c76b6ea771699e3cb4a9034e2be4aa"} Oct 30 13:15:58.989193 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:58.988831 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d" Oct 30 13:15:59.042715 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:15:59.042608 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d" podStartSLOduration=1.654834327 podStartE2EDuration="2.042592886s" totalImagesPullingTime="387.758559ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:59.038026434 +0000 UTC m=+1857.641524880" watchObservedRunningTime="2026-10-30 13:15:59.042592886 +0000 UTC m=+1857.646091331" Oct 30 13:16:05.002159 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:05.002127 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d" Oct 30 13:16:07.353402 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:16:07.353367 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:16:19.709751 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.709695 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8"] Oct 30 13:16:19.711391 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.711361 4931 cpu_manager.go:414] "containerMap: removing container" podUID="01a78c34-1e58-48f3-8660-56d6b19d8847" containerName="cleanup" Oct 30 13:16:19.715164 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.715139 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01a78c34-1e58-48f3-8660-56d6b19d8847" containerName="cleanup" Oct 30 13:16:19.715369 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.715343 4931 cpu_manager.go:414] "containerMap: removing container" podUID="c3bc7915-8ed0-47c5-a693-15db3d153fe8" containerName="cleanup" Oct 30 13:16:19.715369 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.715364 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3bc7915-8ed0-47c5-a693-15db3d153fe8" containerName="cleanup" Oct 30 13:16:19.715576 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.715377 4931 cpu_manager.go:414] "containerMap: removing container" podUID="ea3c064a-c226-4eec-bd7e-d42fb03965ad" containerName="cleanup" Oct 30 13:16:19.715576 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.715385 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea3c064a-c226-4eec-bd7e-d42fb03965ad" containerName="cleanup" Oct 30 13:16:19.715576 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.715452 4931 cpu_manager.go:414] "containerMap: removing container" podUID="2a5f8729-df67-4e02-b7c5-f2250efa9678" containerName="cleanup" Oct 30 13:16:19.715576 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.715460 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a5f8729-df67-4e02-b7c5-f2250efa9678" containerName="cleanup" Oct 30 13:16:19.715576 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.715507 4931 cpu_manager.go:414] "containerMap: removing container" podUID="795f8222-5020-4ef8-be21-49fbb8b178e8" containerName="cleanup" Oct 30 13:16:19.715576 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.715515 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="795f8222-5020-4ef8-be21-49fbb8b178e8" containerName="cleanup" Oct 30 13:16:19.715576 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.715531 4931 cpu_manager.go:414] "containerMap: removing container" podUID="08e53b66-2198-4d37-82d5-563f36393d67" containerName="cleanup" Oct 30 13:16:19.715576 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.715539 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08e53b66-2198-4d37-82d5-563f36393d67" containerName="cleanup" Oct 30 13:16:19.715858 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.715583 4931 cpu_manager.go:414] "containerMap: removing container" podUID="f06e3aa4-5df5-45c3-a80e-02b922c62768" containerName="cleanup" Oct 30 13:16:19.715858 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.715591 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f06e3aa4-5df5-45c3-a80e-02b922c62768" containerName="cleanup" Oct 30 13:16:19.715858 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.715774 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec2e9458-537c-428a-abbe-6ff129af3c6e" containerName="cleanup" Oct 30 13:16:19.715858 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.715787 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="08e53b66-2198-4d37-82d5-563f36393d67" containerName="cleanup" Oct 30 13:16:19.715858 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.715801 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f06e3aa4-5df5-45c3-a80e-02b922c62768" containerName="cleanup" Oct 30 13:16:19.715858 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.715812 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3bc7915-8ed0-47c5-a693-15db3d153fe8" containerName="cleanup" Oct 30 13:16:19.715858 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.715822 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea3c064a-c226-4eec-bd7e-d42fb03965ad" containerName="cleanup" Oct 30 13:16:19.722575 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.722545 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8" Oct 30 13:16:19.739546 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.739372 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8"] Oct 30 13:16:19.746520 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.746485 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2a3e01\"/\"default-dockercfg-pqjvs\"" Oct 30 13:16:19.746520 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.746487 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2a3e01\"/\"openshift-service-ca.crt\"" Oct 30 13:16:19.746743 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.746524 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2a3e01\"/\"kube-root-ca.crt\"" Oct 30 13:16:19.755573 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.755549 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2a3e01\"/\"test-model-2a3e01-kserve-self-signed-certs\"" Oct 30 13:16:19.844284 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.844227 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/844f8313-ec50-42d9-9f6d-7a0467b7a95b-tmp-dir\") pod \"test-model-2a3e01-kserve-54dbfc6f9-c84l8\" (UID: \"844f8313-ec50-42d9-9f6d-7a0467b7a95b\") " pod="ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8" Oct 30 13:16:19.844586 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.844499 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/844f8313-ec50-42d9-9f6d-7a0467b7a95b-dshm\") pod \"test-model-2a3e01-kserve-54dbfc6f9-c84l8\" (UID: \"844f8313-ec50-42d9-9f6d-7a0467b7a95b\") " pod="ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8" Oct 30 13:16:19.844672 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.844587 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/844f8313-ec50-42d9-9f6d-7a0467b7a95b-home\") pod \"test-model-2a3e01-kserve-54dbfc6f9-c84l8\" (UID: \"844f8313-ec50-42d9-9f6d-7a0467b7a95b\") " pod="ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8" Oct 30 13:16:19.844672 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.844653 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/844f8313-ec50-42d9-9f6d-7a0467b7a95b-model-cache\") pod \"test-model-2a3e01-kserve-54dbfc6f9-c84l8\" (UID: \"844f8313-ec50-42d9-9f6d-7a0467b7a95b\") " pod="ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8" Oct 30 13:16:19.844784 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.844723 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lxfj\" (UniqueName: \"kubernetes.io/projected/844f8313-ec50-42d9-9f6d-7a0467b7a95b-kube-api-access-9lxfj\") pod \"test-model-2a3e01-kserve-54dbfc6f9-c84l8\" (UID: \"844f8313-ec50-42d9-9f6d-7a0467b7a95b\") " pod="ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8" Oct 30 13:16:19.844784 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.844749 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/844f8313-ec50-42d9-9f6d-7a0467b7a95b-tls-certs\") pod \"test-model-2a3e01-kserve-54dbfc6f9-c84l8\" (UID: \"844f8313-ec50-42d9-9f6d-7a0467b7a95b\") " pod="ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8" Oct 30 13:16:19.946111 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.946061 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/844f8313-ec50-42d9-9f6d-7a0467b7a95b-dshm\") pod \"test-model-2a3e01-kserve-54dbfc6f9-c84l8\" (UID: \"844f8313-ec50-42d9-9f6d-7a0467b7a95b\") " pod="ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8" Oct 30 13:16:19.946326 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.946151 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/844f8313-ec50-42d9-9f6d-7a0467b7a95b-home\") pod \"test-model-2a3e01-kserve-54dbfc6f9-c84l8\" (UID: \"844f8313-ec50-42d9-9f6d-7a0467b7a95b\") " pod="ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8" Oct 30 13:16:19.946326 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.946204 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/844f8313-ec50-42d9-9f6d-7a0467b7a95b-model-cache\") pod \"test-model-2a3e01-kserve-54dbfc6f9-c84l8\" (UID: \"844f8313-ec50-42d9-9f6d-7a0467b7a95b\") " pod="ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8" Oct 30 13:16:19.946326 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.946256 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lxfj\" (UniqueName: \"kubernetes.io/projected/844f8313-ec50-42d9-9f6d-7a0467b7a95b-kube-api-access-9lxfj\") pod \"test-model-2a3e01-kserve-54dbfc6f9-c84l8\" (UID: \"844f8313-ec50-42d9-9f6d-7a0467b7a95b\") " pod="ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8" Oct 30 13:16:19.946326 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.946302 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/844f8313-ec50-42d9-9f6d-7a0467b7a95b-tls-certs\") pod \"test-model-2a3e01-kserve-54dbfc6f9-c84l8\" (UID: \"844f8313-ec50-42d9-9f6d-7a0467b7a95b\") " pod="ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8" Oct 30 13:16:19.946517 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.946501 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/844f8313-ec50-42d9-9f6d-7a0467b7a95b-tmp-dir\") pod \"test-model-2a3e01-kserve-54dbfc6f9-c84l8\" (UID: \"844f8313-ec50-42d9-9f6d-7a0467b7a95b\") " pod="ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8" Oct 30 13:16:19.947050 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.946657 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/844f8313-ec50-42d9-9f6d-7a0467b7a95b-model-cache\") pod \"test-model-2a3e01-kserve-54dbfc6f9-c84l8\" (UID: \"844f8313-ec50-42d9-9f6d-7a0467b7a95b\") " pod="ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8" Oct 30 13:16:19.947050 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.946888 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/844f8313-ec50-42d9-9f6d-7a0467b7a95b-tmp-dir\") pod \"test-model-2a3e01-kserve-54dbfc6f9-c84l8\" (UID: \"844f8313-ec50-42d9-9f6d-7a0467b7a95b\") " pod="ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8" Oct 30 13:16:19.947990 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.947965 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/844f8313-ec50-42d9-9f6d-7a0467b7a95b-home\") pod \"test-model-2a3e01-kserve-54dbfc6f9-c84l8\" (UID: \"844f8313-ec50-42d9-9f6d-7a0467b7a95b\") " pod="ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8" Oct 30 13:16:19.948960 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.948936 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/844f8313-ec50-42d9-9f6d-7a0467b7a95b-dshm\") pod \"test-model-2a3e01-kserve-54dbfc6f9-c84l8\" (UID: \"844f8313-ec50-42d9-9f6d-7a0467b7a95b\") " pod="ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8" Oct 30 13:16:19.967501 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.949722 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/844f8313-ec50-42d9-9f6d-7a0467b7a95b-tls-certs\") pod \"test-model-2a3e01-kserve-54dbfc6f9-c84l8\" (UID: \"844f8313-ec50-42d9-9f6d-7a0467b7a95b\") " pod="ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8" Oct 30 13:16:19.985326 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:19.985290 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lxfj\" (UniqueName: \"kubernetes.io/projected/844f8313-ec50-42d9-9f6d-7a0467b7a95b-kube-api-access-9lxfj\") pod \"test-model-2a3e01-kserve-54dbfc6f9-c84l8\" (UID: \"844f8313-ec50-42d9-9f6d-7a0467b7a95b\") " pod="ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8" Oct 30 13:16:20.057288 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:20.055976 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8" Oct 30 13:16:20.453634 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:20.453604 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8"] Oct 30 13:16:20.453910 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:16:20.453880 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod844f8313_ec50_42d9_9f6d_7a0467b7a95b.slice/crio-f513088405425286b33e3dc6b2405c30a69a5f06e4d54cf465e54d56b9a48f15 WatchSource:0}: Error finding container f513088405425286b33e3dc6b2405c30a69a5f06e4d54cf465e54d56b9a48f15: Status 404 returned error can't find the container with id f513088405425286b33e3dc6b2405c30a69a5f06e4d54cf465e54d56b9a48f15 Oct 30 13:16:21.198290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:21.195160 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8" event={"ID":"844f8313-ec50-42d9-9f6d-7a0467b7a95b","Type":"ContainerStarted","Data":"96403e9d198cce9795668c3885edd7f8ccb1b9f516ae69126d6ad46098efd1f0"} Oct 30 13:16:21.198290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:21.195224 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8" event={"ID":"844f8313-ec50-42d9-9f6d-7a0467b7a95b","Type":"ContainerStarted","Data":"f513088405425286b33e3dc6b2405c30a69a5f06e4d54cf465e54d56b9a48f15"} Oct 30 13:16:21.266371 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:21.265705 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8" podStartSLOduration=2.265684982 podStartE2EDuration="2.265684982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:21.256790388 +0000 UTC m=+1879.860288857" watchObservedRunningTime="2026-10-30 13:16:21.265684982 +0000 UTC m=+1879.869183429" Oct 30 13:16:22.017128 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:22.017096 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b9e8e-d75gd"] Oct 30 13:16:22.030118 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:22.030087 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b9e8e-d75gd"] Oct 30 13:16:22.057633 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:22.057604 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8" Oct 30 13:16:23.514686 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:23.514652 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29227526-2318-4d64-8423-bb63666f80c7" path="/var/lib/kubelet/pods/29227526-2318-4d64-8423-bb63666f80c7/volumes" Oct 30 13:16:27.604394 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:27.604344 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:16:28.133338 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:28.124036 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh"] Oct 30 13:16:28.133338 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:28.124377 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" podUID="16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed" containerName="istio-proxy" containerID="cri-o://08442ed9c67d02f34f5c98e5886ef7b41e83df17b029b435c8d57ddfbc2dd4b0" gracePeriod=30 Oct 30 13:16:28.137414 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:28.137375 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8"] Oct 30 13:16:28.137754 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:28.137715 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" podUID="4e1ed676-cd86-4b17-97af-1a5a501f6274" containerName="istio-proxy" containerID="cri-o://38fa2303b37360ec91dc36073c63945b29064d92b92fab921da816d78b386456" gracePeriod=30 Oct 30 13:16:30.058128 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:30.058092 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8" Oct 30 13:16:30.071143 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:30.071090 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8" Oct 30 13:16:30.292796 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:30.292760 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8" Oct 30 13:16:32.133713 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:32.133679 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14e8a4-z4jfb"] Oct 30 13:16:32.138499 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:32.138476 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14e8a4-z4jfb" Oct 30 13:16:32.162515 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:32.162487 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:16:32.171671 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:32.171649 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:16:32.173891 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:32.173865 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-14e8a4-z4jfb\" (UID: \"f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14e8a4-z4jfb" Oct 30 13:16:32.174001 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:32.173915 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6xq9\" (UniqueName: \"kubernetes.io/projected/f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1-kube-api-access-t6xq9\") pod \"maas-api-revoke-keys-e2e-api-b-14e8a4-z4jfb\" (UID: \"f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14e8a4-z4jfb" Oct 30 13:16:32.174166 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:32.174147 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-14e8a4-z4jfb\" (UID: \"f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14e8a4-z4jfb" Oct 30 13:16:32.188595 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:32.188567 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14e8a4-z4jfb"] Oct 30 13:16:32.276318 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:32.276259 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-14e8a4-z4jfb\" (UID: \"f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14e8a4-z4jfb" Oct 30 13:16:32.276694 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:32.276670 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-14e8a4-z4jfb\" (UID: \"f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14e8a4-z4jfb" Oct 30 13:16:32.276836 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:32.276820 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6xq9\" (UniqueName: \"kubernetes.io/projected/f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1-kube-api-access-t6xq9\") pod \"maas-api-revoke-keys-e2e-api-b-14e8a4-z4jfb\" (UID: \"f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14e8a4-z4jfb" Oct 30 13:16:32.279700 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:32.279664 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-14e8a4-z4jfb\" (UID: \"f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14e8a4-z4jfb" Oct 30 13:16:32.289363 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:32.289320 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-14e8a4-z4jfb\" (UID: \"f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14e8a4-z4jfb" Oct 30 13:16:32.304128 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:32.304100 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6xq9\" (UniqueName: \"kubernetes.io/projected/f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1-kube-api-access-t6xq9\") pod \"maas-api-revoke-keys-e2e-api-b-14e8a4-z4jfb\" (UID: \"f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14e8a4-z4jfb" Oct 30 13:16:32.453384 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:32.453353 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14e8a4-z4jfb" Oct 30 13:16:32.770660 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:32.770625 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14e8a4-z4jfb"] Oct 30 13:16:32.770868 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:16:32.770808 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7ce9e0a_5781_41a8_8c60_98fa9ecce6b1.slice/crio-e1facaf811ab4a737c4b2fcea0fbb45e1749038d4a646cfe8e16f2e57964f7f6 WatchSource:0}: Error finding container e1facaf811ab4a737c4b2fcea0fbb45e1749038d4a646cfe8e16f2e57964f7f6: Status 404 returned error can't find the container with id e1facaf811ab4a737c4b2fcea0fbb45e1749038d4a646cfe8e16f2e57964f7f6 Oct 30 13:16:33.328666 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.327585 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14e8a4-z4jfb" event={"ID":"f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1","Type":"ContainerStarted","Data":"58dfec2f3c35b81d7f0b2d69e47880ef34d27afdff8a2fca0aa4a42d39e60e55"} Oct 30 13:16:33.328666 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.327635 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14e8a4-z4jfb" event={"ID":"f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1","Type":"ContainerStarted","Data":"e1facaf811ab4a737c4b2fcea0fbb45e1749038d4a646cfe8e16f2e57964f7f6"} Oct 30 13:16:33.331088 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.330898 4931 generic.go:356] "Generic (PLEG): container finished" podID="4e1ed676-cd86-4b17-97af-1a5a501f6274" containerID="38fa2303b37360ec91dc36073c63945b29064d92b92fab921da816d78b386456" exitCode=0 Oct 30 13:16:33.331088 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.331005 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" event={"ID":"4e1ed676-cd86-4b17-97af-1a5a501f6274","Type":"ContainerDied","Data":"38fa2303b37360ec91dc36073c63945b29064d92b92fab921da816d78b386456"} Oct 30 13:16:33.333282 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.333065 4931 generic.go:356] "Generic (PLEG): container finished" podID="16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed" containerID="08442ed9c67d02f34f5c98e5886ef7b41e83df17b029b435c8d57ddfbc2dd4b0" exitCode=0 Oct 30 13:16:33.333282 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.333213 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" event={"ID":"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed","Type":"ContainerDied","Data":"08442ed9c67d02f34f5c98e5886ef7b41e83df17b029b435c8d57ddfbc2dd4b0"} Oct 30 13:16:33.542000 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.541952 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" Oct 30 13:16:33.605851 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.605820 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" Oct 30 13:16:33.611135 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.611082 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14e8a4-z4jfb" podStartSLOduration=1.611064803 podStartE2EDuration="1.611064803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:33.413793481 +0000 UTC m=+1892.017291929" watchObservedRunningTime="2026-10-30 13:16:33.611064803 +0000 UTC m=+1892.214563250" Oct 30 13:16:33.613106 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.612452 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e1ed676-cd86-4b17-97af-1a5a501f6274-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e1ed676-cd86-4b17-97af-1a5a501f6274-istio-data\") pod \"4e1ed676-cd86-4b17-97af-1a5a501f6274\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " Oct 30 13:16:33.613106 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.612509 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e1ed676-cd86-4b17-97af-1a5a501f6274-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1ed676-cd86-4b17-97af-1a5a501f6274-workload-socket\") pod \"4e1ed676-cd86-4b17-97af-1a5a501f6274\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " Oct 30 13:16:33.613106 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.612627 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e1ed676-cd86-4b17-97af-1a5a501f6274-kube-api-access-wznc8\" (UniqueName: \"kubernetes.io/projected/4e1ed676-cd86-4b17-97af-1a5a501f6274-kube-api-access-wznc8\") pod \"4e1ed676-cd86-4b17-97af-1a5a501f6274\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " Oct 30 13:16:33.613106 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.612657 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e1ed676-cd86-4b17-97af-1a5a501f6274-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e1ed676-cd86-4b17-97af-1a5a501f6274-istio-envoy\") pod \"4e1ed676-cd86-4b17-97af-1a5a501f6274\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " Oct 30 13:16:33.613106 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.612750 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e1ed676-cd86-4b17-97af-1a5a501f6274-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e1ed676-cd86-4b17-97af-1a5a501f6274-istiod-ca-cert\") pod \"4e1ed676-cd86-4b17-97af-1a5a501f6274\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " Oct 30 13:16:33.613106 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.612896 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e1ed676-cd86-4b17-97af-1a5a501f6274-istio-token\" (UniqueName: \"kubernetes.io/projected/4e1ed676-cd86-4b17-97af-1a5a501f6274-istio-token\") pod \"4e1ed676-cd86-4b17-97af-1a5a501f6274\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " Oct 30 13:16:33.613106 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.612937 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e1ed676-cd86-4b17-97af-1a5a501f6274-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e1ed676-cd86-4b17-97af-1a5a501f6274-istio-podinfo\") pod \"4e1ed676-cd86-4b17-97af-1a5a501f6274\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " Oct 30 13:16:33.613106 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.612973 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e1ed676-cd86-4b17-97af-1a5a501f6274-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1ed676-cd86-4b17-97af-1a5a501f6274-credential-socket\") pod \"4e1ed676-cd86-4b17-97af-1a5a501f6274\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " Oct 30 13:16:33.613106 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.613022 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e1ed676-cd86-4b17-97af-1a5a501f6274-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e1ed676-cd86-4b17-97af-1a5a501f6274-workload-certs\") pod \"4e1ed676-cd86-4b17-97af-1a5a501f6274\" (UID: \"4e1ed676-cd86-4b17-97af-1a5a501f6274\") " Oct 30 13:16:33.614327 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.614245 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e1ed676-cd86-4b17-97af-1a5a501f6274-istio-data" pod "4e1ed676-cd86-4b17-97af-1a5a501f6274" (UID: "4e1ed676-cd86-4b17-97af-1a5a501f6274"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:33.614452 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.614410 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e1ed676-cd86-4b17-97af-1a5a501f6274-istiod-ca-cert" pod "4e1ed676-cd86-4b17-97af-1a5a501f6274" (UID: "4e1ed676-cd86-4b17-97af-1a5a501f6274"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:16:33.614780 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.614613 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e1ed676-cd86-4b17-97af-1a5a501f6274-credential-socket" pod "4e1ed676-cd86-4b17-97af-1a5a501f6274" (UID: "4e1ed676-cd86-4b17-97af-1a5a501f6274"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:33.615196 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.615175 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e1ed676-cd86-4b17-97af-1a5a501f6274-workload-certs" pod "4e1ed676-cd86-4b17-97af-1a5a501f6274" (UID: "4e1ed676-cd86-4b17-97af-1a5a501f6274"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:33.615472 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.615445 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e1ed676-cd86-4b17-97af-1a5a501f6274-workload-socket" pod "4e1ed676-cd86-4b17-97af-1a5a501f6274" (UID: "4e1ed676-cd86-4b17-97af-1a5a501f6274"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:33.616615 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.616585 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e1ed676-cd86-4b17-97af-1a5a501f6274-istio-token" pod "4e1ed676-cd86-4b17-97af-1a5a501f6274" (UID: "4e1ed676-cd86-4b17-97af-1a5a501f6274"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:33.616721 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.616624 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e1ed676-cd86-4b17-97af-1a5a501f6274-istio-podinfo" pod "4e1ed676-cd86-4b17-97af-1a5a501f6274" (UID: "4e1ed676-cd86-4b17-97af-1a5a501f6274"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:16:33.617157 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.617128 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e1ed676-cd86-4b17-97af-1a5a501f6274-istio-envoy" pod "4e1ed676-cd86-4b17-97af-1a5a501f6274" (UID: "4e1ed676-cd86-4b17-97af-1a5a501f6274"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:33.617453 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.617433 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e1ed676-cd86-4b17-97af-1a5a501f6274-kube-api-access-wznc8" pod "4e1ed676-cd86-4b17-97af-1a5a501f6274" (UID: "4e1ed676-cd86-4b17-97af-1a5a501f6274"). InnerVolumeSpecName "kube-api-access-wznc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:33.715019 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.714941 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-credential-socket\") pod \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " Oct 30 13:16:33.715019 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.714998 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-kube-api-access-c8s68\" (UniqueName: \"kubernetes.io/projected/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-kube-api-access-c8s68\") pod \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " Oct 30 13:16:33.715248 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.715053 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-workload-socket\") pod \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " Oct 30 13:16:33.715248 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.715176 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-istio-data\") pod \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " Oct 30 13:16:33.715389 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.715245 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-credential-socket" pod "16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed" (UID: "16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:33.715389 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.715352 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-istiod-ca-cert\") pod \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " Oct 30 13:16:33.715497 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.715390 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-workload-socket" pod "16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed" (UID: "16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:33.715497 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.715417 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-istio-podinfo\") pod \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " Oct 30 13:16:33.715497 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.715425 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-istio-data" pod "16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed" (UID: "16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:33.715650 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.715542 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-istio-token\" (UniqueName: \"kubernetes.io/projected/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-istio-token\") pod \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " Oct 30 13:16:33.715650 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.715592 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-istio-envoy\") pod \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " Oct 30 13:16:33.715650 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.715640 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-workload-certs\") pod \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\" (UID: \"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed\") " Oct 30 13:16:33.715805 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.715669 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-istiod-ca-cert" pod "16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed" (UID: "16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:16:33.716007 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.715978 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-workload-certs" pod "16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed" (UID: "16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:33.716689 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.716662 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e1ed676-cd86-4b17-97af-1a5a501f6274-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:16:33.716815 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.716692 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:16:33.716815 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.716707 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1ed676-cd86-4b17-97af-1a5a501f6274-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:16:33.716815 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.716722 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:16:33.716815 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.716738 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wznc8\" (UniqueName: \"kubernetes.io/projected/4e1ed676-cd86-4b17-97af-1a5a501f6274-kube-api-access-wznc8\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:16:33.716815 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.716751 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e1ed676-cd86-4b17-97af-1a5a501f6274-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:16:33.716815 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.716766 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e1ed676-cd86-4b17-97af-1a5a501f6274-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:16:33.716815 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.716779 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:16:33.716815 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.716791 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e1ed676-cd86-4b17-97af-1a5a501f6274-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:16:33.716815 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.716804 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e1ed676-cd86-4b17-97af-1a5a501f6274-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:16:33.716815 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.716817 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1ed676-cd86-4b17-97af-1a5a501f6274-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:16:33.717437 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.716830 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e1ed676-cd86-4b17-97af-1a5a501f6274-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:16:33.717437 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.716844 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:16:33.717437 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.716856 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:16:33.717658 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.717636 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-istio-podinfo" pod "16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed" (UID: "16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:16:33.717795 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.717776 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-istio-envoy" pod "16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed" (UID: "16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:33.717858 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.717782 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-kube-api-access-c8s68" pod "16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed" (UID: "16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed"). InnerVolumeSpecName "kube-api-access-c8s68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:33.717980 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.717963 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-istio-token" pod "16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed" (UID: "16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:33.818849 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.818819 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:16:33.818849 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.818845 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:16:33.818849 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.818854 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8s68\" (UniqueName: \"kubernetes.io/projected/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-kube-api-access-c8s68\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:16:33.819108 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:33.818868 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:16:34.353073 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:34.353036 4931 generic.go:356] "Generic (PLEG): container finished" podID="f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1" containerID="58dfec2f3c35b81d7f0b2d69e47880ef34d27afdff8a2fca0aa4a42d39e60e55" exitCode=0 Oct 30 13:16:34.353576 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:34.353124 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14e8a4-z4jfb" event={"ID":"f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1","Type":"ContainerDied","Data":"58dfec2f3c35b81d7f0b2d69e47880ef34d27afdff8a2fca0aa4a42d39e60e55"} Oct 30 13:16:34.354657 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:34.354633 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" Oct 30 13:16:34.354657 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:34.354637 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8" event={"ID":"4e1ed676-cd86-4b17-97af-1a5a501f6274","Type":"ContainerDied","Data":"f67d6ab3a49497ea1b6aad79a42570e0a5b34fc91c0ed758fb7f3eeaed8452db"} Oct 30 13:16:34.354853 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:34.354695 4931 scope.go:122] "RemoveContainer" containerID="38fa2303b37360ec91dc36073c63945b29064d92b92fab921da816d78b386456" Oct 30 13:16:34.356332 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:34.356309 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" event={"ID":"16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed","Type":"ContainerDied","Data":"4be40406fef651a76fc35884a2b0fd884110373168dd4ce7e2561f73b6a1868e"} Oct 30 13:16:34.356420 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:34.356338 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" Oct 30 13:16:34.366770 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:34.366667 4931 scope.go:122] "RemoveContainer" containerID="08442ed9c67d02f34f5c98e5886ef7b41e83df17b029b435c8d57ddfbc2dd4b0" Oct 30 13:16:34.480103 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:34.480072 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh"] Oct 30 13:16:34.493436 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:34.493402 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh"] Oct 30 13:16:34.529408 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:34.529379 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8"] Oct 30 13:16:34.542303 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:34.542254 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8"] Oct 30 13:16:34.546553 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:34.546481 4931 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:16:34.546678 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:34.546552 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh" podUID="16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:16:35.515735 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:35.515705 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed" path="/var/lib/kubelet/pods/16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed/volumes" Oct 30 13:16:35.516261 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:35.516239 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e1ed676-cd86-4b17-97af-1a5a501f6274" path="/var/lib/kubelet/pods/4e1ed676-cd86-4b17-97af-1a5a501f6274/volumes" Oct 30 13:16:35.531825 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:35.531803 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14e8a4-z4jfb" Oct 30 13:16:35.642433 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:35.642345 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1-maas-api-xks-ca\") pod \"f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1\" (UID: \"f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1\") " Oct 30 13:16:35.642433 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:35.642423 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1-kube-api-access-t6xq9\" (UniqueName: \"kubernetes.io/projected/f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1-kube-api-access-t6xq9\") pod \"f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1\" (UID: \"f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1\") " Oct 30 13:16:35.642750 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:35.642734 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1-maas-api-openshift-ca\") pod \"f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1\" (UID: \"f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1\") " Oct 30 13:16:35.643119 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:35.643081 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1-maas-api-openshift-ca" pod "f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1" (UID: "f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:16:35.643355 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:35.643336 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:16:35.644719 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:35.644693 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1-maas-api-xks-ca" pod "f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1" (UID: "f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:16:35.644846 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:35.644694 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1-kube-api-access-t6xq9" pod "f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1" (UID: "f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1"). InnerVolumeSpecName "kube-api-access-t6xq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:35.745009 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:35.744976 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6xq9\" (UniqueName: \"kubernetes.io/projected/f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1-kube-api-access-t6xq9\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:16:35.745009 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:35.745002 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:16:36.376360 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:36.376329 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14e8a4-z4jfb" Oct 30 13:16:36.376569 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:36.376334 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14e8a4-z4jfb" event={"ID":"f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1","Type":"ContainerDied","Data":"e1facaf811ab4a737c4b2fcea0fbb45e1749038d4a646cfe8e16f2e57964f7f6"} Oct 30 13:16:36.376569 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:36.376456 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1facaf811ab4a737c4b2fcea0fbb45e1749038d4a646cfe8e16f2e57964f7f6" Oct 30 13:16:37.352795 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:16:37.352761 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:16:42.876752 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:42.876713 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-11cfa6-775hp"] Oct 30 13:16:42.877338 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:42.877323 4931 cpu_manager.go:414] "containerMap: removing container" podUID="16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed" containerName="istio-proxy" Oct 30 13:16:42.877338 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:42.877338 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed" containerName="istio-proxy" Oct 30 13:16:42.877446 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:42.877350 4931 cpu_manager.go:414] "containerMap: removing container" podUID="4e1ed676-cd86-4b17-97af-1a5a501f6274" containerName="istio-proxy" Oct 30 13:16:42.877446 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:42.877355 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e1ed676-cd86-4b17-97af-1a5a501f6274" containerName="istio-proxy" Oct 30 13:16:42.877446 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:42.877396 4931 cpu_manager.go:414] "containerMap: removing container" podUID="f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1" containerName="revoke-keys" Oct 30 13:16:42.877446 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:42.877401 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1" containerName="revoke-keys" Oct 30 13:16:42.877627 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:42.877484 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1" containerName="revoke-keys" Oct 30 13:16:42.877627 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:42.877495 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="16d1bbe1-f3d7-4670-9bc7-b27dfe1e9eed" containerName="istio-proxy" Oct 30 13:16:42.877627 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:42.877502 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e1ed676-cd86-4b17-97af-1a5a501f6274" containerName="istio-proxy" Oct 30 13:16:42.881320 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:42.881305 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-11cfa6-775hp" Oct 30 13:16:42.910163 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:42.906939 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-11cfa6-775hp"] Oct 30 13:16:42.913951 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:42.913921 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:16:42.914091 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:42.914026 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:16:42.963153 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:42.963111 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f99mm\" (UniqueName: \"kubernetes.io/projected/4eba97ae-a62f-4ae2-bc04-e28404c7dc21-kube-api-access-f99mm\") pod \"maas-api-revoke-keys-e2e-api-b-11cfa6-775hp\" (UID: \"4eba97ae-a62f-4ae2-bc04-e28404c7dc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-11cfa6-775hp" Oct 30 13:16:42.963414 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:42.963361 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eba97ae-a62f-4ae2-bc04-e28404c7dc21-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-11cfa6-775hp\" (UID: \"4eba97ae-a62f-4ae2-bc04-e28404c7dc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-11cfa6-775hp" Oct 30 13:16:42.963568 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:42.963495 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eba97ae-a62f-4ae2-bc04-e28404c7dc21-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-11cfa6-775hp\" (UID: \"4eba97ae-a62f-4ae2-bc04-e28404c7dc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-11cfa6-775hp" Oct 30 13:16:43.064801 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:43.064762 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f99mm\" (UniqueName: \"kubernetes.io/projected/4eba97ae-a62f-4ae2-bc04-e28404c7dc21-kube-api-access-f99mm\") pod \"maas-api-revoke-keys-e2e-api-b-11cfa6-775hp\" (UID: \"4eba97ae-a62f-4ae2-bc04-e28404c7dc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-11cfa6-775hp" Oct 30 13:16:43.064998 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:43.064869 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eba97ae-a62f-4ae2-bc04-e28404c7dc21-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-11cfa6-775hp\" (UID: \"4eba97ae-a62f-4ae2-bc04-e28404c7dc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-11cfa6-775hp" Oct 30 13:16:43.064998 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:43.064920 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eba97ae-a62f-4ae2-bc04-e28404c7dc21-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-11cfa6-775hp\" (UID: \"4eba97ae-a62f-4ae2-bc04-e28404c7dc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-11cfa6-775hp" Oct 30 13:16:43.065705 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:43.065678 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eba97ae-a62f-4ae2-bc04-e28404c7dc21-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-11cfa6-775hp\" (UID: \"4eba97ae-a62f-4ae2-bc04-e28404c7dc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-11cfa6-775hp" Oct 30 13:16:43.067480 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:43.067452 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eba97ae-a62f-4ae2-bc04-e28404c7dc21-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-11cfa6-775hp\" (UID: \"4eba97ae-a62f-4ae2-bc04-e28404c7dc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-11cfa6-775hp" Oct 30 13:16:43.098671 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:43.098646 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f99mm\" (UniqueName: \"kubernetes.io/projected/4eba97ae-a62f-4ae2-bc04-e28404c7dc21-kube-api-access-f99mm\") pod \"maas-api-revoke-keys-e2e-api-b-11cfa6-775hp\" (UID: \"4eba97ae-a62f-4ae2-bc04-e28404c7dc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-11cfa6-775hp" Oct 30 13:16:43.200786 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:43.200759 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-11cfa6-775hp" Oct 30 13:16:43.534815 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:43.534771 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-11cfa6-775hp"] Oct 30 13:16:43.535050 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:16:43.535018 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eba97ae_a62f_4ae2_bc04_e28404c7dc21.slice/crio-c2b66eb860d10c18e6ba189cc8771f2e31565236e1bd41ac2f4e95791446c3b2 WatchSource:0}: Error finding container c2b66eb860d10c18e6ba189cc8771f2e31565236e1bd41ac2f4e95791446c3b2: Status 404 returned error can't find the container with id c2b66eb860d10c18e6ba189cc8771f2e31565236e1bd41ac2f4e95791446c3b2 Oct 30 13:16:44.514797 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:44.514764 4931 generic.go:356] "Generic (PLEG): container finished" podID="4eba97ae-a62f-4ae2-bc04-e28404c7dc21" containerID="05b1d0f687144e9951b6e4f7bd79cd5fbe3afd8d57ef4ec810e651a6efb70aa5" exitCode=0 Oct 30 13:16:44.515201 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:44.514847 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-11cfa6-775hp" event={"ID":"4eba97ae-a62f-4ae2-bc04-e28404c7dc21","Type":"ContainerDied","Data":"05b1d0f687144e9951b6e4f7bd79cd5fbe3afd8d57ef4ec810e651a6efb70aa5"} Oct 30 13:16:44.515201 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:44.514912 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-11cfa6-775hp" event={"ID":"4eba97ae-a62f-4ae2-bc04-e28404c7dc21","Type":"ContainerStarted","Data":"c2b66eb860d10c18e6ba189cc8771f2e31565236e1bd41ac2f4e95791446c3b2"} Oct 30 13:16:45.692604 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:45.692576 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-11cfa6-775hp" Oct 30 13:16:45.814428 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:45.814354 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eba97ae-a62f-4ae2-bc04-e28404c7dc21-kube-api-access-f99mm\" (UniqueName: \"kubernetes.io/projected/4eba97ae-a62f-4ae2-bc04-e28404c7dc21-kube-api-access-f99mm\") pod \"4eba97ae-a62f-4ae2-bc04-e28404c7dc21\" (UID: \"4eba97ae-a62f-4ae2-bc04-e28404c7dc21\") " Oct 30 13:16:45.814428 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:45.814418 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eba97ae-a62f-4ae2-bc04-e28404c7dc21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eba97ae-a62f-4ae2-bc04-e28404c7dc21-maas-api-openshift-ca\") pod \"4eba97ae-a62f-4ae2-bc04-e28404c7dc21\" (UID: \"4eba97ae-a62f-4ae2-bc04-e28404c7dc21\") " Oct 30 13:16:45.814705 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:45.814666 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eba97ae-a62f-4ae2-bc04-e28404c7dc21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eba97ae-a62f-4ae2-bc04-e28404c7dc21-maas-api-xks-ca\") pod \"4eba97ae-a62f-4ae2-bc04-e28404c7dc21\" (UID: \"4eba97ae-a62f-4ae2-bc04-e28404c7dc21\") " Oct 30 13:16:45.814835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:45.814810 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eba97ae-a62f-4ae2-bc04-e28404c7dc21-maas-api-openshift-ca" pod "4eba97ae-a62f-4ae2-bc04-e28404c7dc21" (UID: "4eba97ae-a62f-4ae2-bc04-e28404c7dc21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:16:45.815292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:45.815258 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eba97ae-a62f-4ae2-bc04-e28404c7dc21-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.816745 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:45.816722 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eba97ae-a62f-4ae2-bc04-e28404c7dc21-maas-api-xks-ca" pod "4eba97ae-a62f-4ae2-bc04-e28404c7dc21" (UID: "4eba97ae-a62f-4ae2-bc04-e28404c7dc21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:16:45.816888 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:45.816867 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eba97ae-a62f-4ae2-bc04-e28404c7dc21-kube-api-access-f99mm" pod "4eba97ae-a62f-4ae2-bc04-e28404c7dc21" (UID: "4eba97ae-a62f-4ae2-bc04-e28404c7dc21"). InnerVolumeSpecName "kube-api-access-f99mm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:45.917159 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:45.917119 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eba97ae-a62f-4ae2-bc04-e28404c7dc21-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.917159 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:45.917156 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f99mm\" (UniqueName: \"kubernetes.io/projected/4eba97ae-a62f-4ae2-bc04-e28404c7dc21-kube-api-access-f99mm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:16:46.531368 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:46.531340 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-11cfa6-775hp" Oct 30 13:16:46.531536 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:46.531393 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-11cfa6-775hp" event={"ID":"4eba97ae-a62f-4ae2-bc04-e28404c7dc21","Type":"ContainerDied","Data":"c2b66eb860d10c18e6ba189cc8771f2e31565236e1bd41ac2f4e95791446c3b2"} Oct 30 13:16:46.531536 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:46.531448 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2b66eb860d10c18e6ba189cc8771f2e31565236e1bd41ac2f4e95791446c3b2" Oct 30 13:16:51.600787 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:51.600753 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc"] Oct 30 13:16:51.601679 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:51.601655 4931 cpu_manager.go:414] "containerMap: removing container" podUID="4eba97ae-a62f-4ae2-bc04-e28404c7dc21" containerName="revoke-keys" Oct 30 13:16:51.601679 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:51.601679 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eba97ae-a62f-4ae2-bc04-e28404c7dc21" containerName="revoke-keys" Oct 30 13:16:51.604020 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:51.601933 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eba97ae-a62f-4ae2-bc04-e28404c7dc21" containerName="revoke-keys" Oct 30 13:16:51.634293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:51.624530 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc" Oct 30 13:16:51.683810 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:51.683734 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc"] Oct 30 13:16:51.687693 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:51.684397 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-649303\"/\"kube-root-ca.crt\"" Oct 30 13:16:51.687901 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:51.684720 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-649303\"/\"test-model-649303-kserve-self-signed-certs\"" Oct 30 13:16:51.688207 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:51.684813 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-649303\"/\"default-dockercfg-65zj9\"" Oct 30 13:16:51.688367 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:51.684907 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-649303\"/\"openshift-service-ca.crt\"" Oct 30 13:16:51.712441 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:51.712396 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqpfw\" (UniqueName: \"kubernetes.io/projected/823d8e4a-010a-4760-a070-f05bc6c2d7cc-kube-api-access-qqpfw\") pod \"test-model-649303-kserve-6f78cffb4f-hgxxc\" (UID: \"823d8e4a-010a-4760-a070-f05bc6c2d7cc\") " pod="ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc" Oct 30 13:16:51.712868 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:51.712844 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/823d8e4a-010a-4760-a070-f05bc6c2d7cc-model-cache\") pod \"test-model-649303-kserve-6f78cffb4f-hgxxc\" (UID: \"823d8e4a-010a-4760-a070-f05bc6c2d7cc\") " pod="ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc" Oct 30 13:16:51.712994 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:51.712908 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/823d8e4a-010a-4760-a070-f05bc6c2d7cc-dshm\") pod \"test-model-649303-kserve-6f78cffb4f-hgxxc\" (UID: \"823d8e4a-010a-4760-a070-f05bc6c2d7cc\") " pod="ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc" Oct 30 13:16:51.712994 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:51.712966 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/823d8e4a-010a-4760-a070-f05bc6c2d7cc-home\") pod \"test-model-649303-kserve-6f78cffb4f-hgxxc\" (UID: \"823d8e4a-010a-4760-a070-f05bc6c2d7cc\") " pod="ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc" Oct 30 13:16:51.713107 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:51.712992 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/823d8e4a-010a-4760-a070-f05bc6c2d7cc-tls-certs\") pod \"test-model-649303-kserve-6f78cffb4f-hgxxc\" (UID: \"823d8e4a-010a-4760-a070-f05bc6c2d7cc\") " pod="ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc" Oct 30 13:16:51.713107 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:51.713050 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/823d8e4a-010a-4760-a070-f05bc6c2d7cc-tmp-dir\") pod \"test-model-649303-kserve-6f78cffb4f-hgxxc\" (UID: \"823d8e4a-010a-4760-a070-f05bc6c2d7cc\") " pod="ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc" Oct 30 13:16:51.814162 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:51.814121 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/823d8e4a-010a-4760-a070-f05bc6c2d7cc-home\") pod \"test-model-649303-kserve-6f78cffb4f-hgxxc\" (UID: \"823d8e4a-010a-4760-a070-f05bc6c2d7cc\") " pod="ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc" Oct 30 13:16:51.814411 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:51.814174 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/823d8e4a-010a-4760-a070-f05bc6c2d7cc-tls-certs\") pod \"test-model-649303-kserve-6f78cffb4f-hgxxc\" (UID: \"823d8e4a-010a-4760-a070-f05bc6c2d7cc\") " pod="ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc" Oct 30 13:16:51.814411 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:51.814235 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/823d8e4a-010a-4760-a070-f05bc6c2d7cc-tmp-dir\") pod \"test-model-649303-kserve-6f78cffb4f-hgxxc\" (UID: \"823d8e4a-010a-4760-a070-f05bc6c2d7cc\") " pod="ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc" Oct 30 13:16:51.814526 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:51.814416 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqpfw\" (UniqueName: \"kubernetes.io/projected/823d8e4a-010a-4760-a070-f05bc6c2d7cc-kube-api-access-qqpfw\") pod \"test-model-649303-kserve-6f78cffb4f-hgxxc\" (UID: \"823d8e4a-010a-4760-a070-f05bc6c2d7cc\") " pod="ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc" Oct 30 13:16:51.814619 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:51.814598 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/823d8e4a-010a-4760-a070-f05bc6c2d7cc-model-cache\") pod \"test-model-649303-kserve-6f78cffb4f-hgxxc\" (UID: \"823d8e4a-010a-4760-a070-f05bc6c2d7cc\") " pod="ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc" Oct 30 13:16:51.814678 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:51.814607 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/823d8e4a-010a-4760-a070-f05bc6c2d7cc-home\") pod \"test-model-649303-kserve-6f78cffb4f-hgxxc\" (UID: \"823d8e4a-010a-4760-a070-f05bc6c2d7cc\") " pod="ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc" Oct 30 13:16:51.814729 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:51.814685 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/823d8e4a-010a-4760-a070-f05bc6c2d7cc-dshm\") pod \"test-model-649303-kserve-6f78cffb4f-hgxxc\" (UID: \"823d8e4a-010a-4760-a070-f05bc6c2d7cc\") " pod="ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc" Oct 30 13:16:51.814977 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:51.814954 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/823d8e4a-010a-4760-a070-f05bc6c2d7cc-model-cache\") pod \"test-model-649303-kserve-6f78cffb4f-hgxxc\" (UID: \"823d8e4a-010a-4760-a070-f05bc6c2d7cc\") " pod="ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc" Oct 30 13:16:51.815037 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:51.814988 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/823d8e4a-010a-4760-a070-f05bc6c2d7cc-tmp-dir\") pod \"test-model-649303-kserve-6f78cffb4f-hgxxc\" (UID: \"823d8e4a-010a-4760-a070-f05bc6c2d7cc\") " pod="ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc" Oct 30 13:16:51.817208 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:51.817186 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/823d8e4a-010a-4760-a070-f05bc6c2d7cc-tls-certs\") pod \"test-model-649303-kserve-6f78cffb4f-hgxxc\" (UID: \"823d8e4a-010a-4760-a070-f05bc6c2d7cc\") " pod="ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc" Oct 30 13:16:51.821522 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:51.821493 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/823d8e4a-010a-4760-a070-f05bc6c2d7cc-dshm\") pod \"test-model-649303-kserve-6f78cffb4f-hgxxc\" (UID: \"823d8e4a-010a-4760-a070-f05bc6c2d7cc\") " pod="ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc" Oct 30 13:16:51.863086 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:51.863012 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqpfw\" (UniqueName: \"kubernetes.io/projected/823d8e4a-010a-4760-a070-f05bc6c2d7cc-kube-api-access-qqpfw\") pod \"test-model-649303-kserve-6f78cffb4f-hgxxc\" (UID: \"823d8e4a-010a-4760-a070-f05bc6c2d7cc\") " pod="ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc" Oct 30 13:16:52.044292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:52.032387 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc" Oct 30 13:16:52.422092 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:52.422060 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc"] Oct 30 13:16:52.423449 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:16:52.423411 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod823d8e4a_010a_4760_a070_f05bc6c2d7cc.slice/crio-16213195f102cd87ea31bee6642cf466748412eff091fdb57c6c69db1a18b2fc WatchSource:0}: Error finding container 16213195f102cd87ea31bee6642cf466748412eff091fdb57c6c69db1a18b2fc: Status 404 returned error can't find the container with id 16213195f102cd87ea31bee6642cf466748412eff091fdb57c6c69db1a18b2fc Oct 30 13:16:52.618712 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:52.618671 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc" event={"ID":"823d8e4a-010a-4760-a070-f05bc6c2d7cc","Type":"ContainerStarted","Data":"6894ab6f9fb6b26049b84e9844a9e86ea7dcd86a36d74f51885da3c1f3103341"} Oct 30 13:16:52.619194 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:52.618729 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc" event={"ID":"823d8e4a-010a-4760-a070-f05bc6c2d7cc","Type":"ContainerStarted","Data":"16213195f102cd87ea31bee6642cf466748412eff091fdb57c6c69db1a18b2fc"} Oct 30 13:16:52.675512 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:52.675415 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc" podStartSLOduration=1.675398912 podStartE2EDuration="1.675398912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:52.670160427 +0000 UTC m=+1911.273658874" watchObservedRunningTime="2026-10-30 13:16:52.675398912 +0000 UTC m=+1911.278897358" Oct 30 13:16:53.033231 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:53.033198 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc" Oct 30 13:16:57.859745 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:57.859707 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz"] Oct 30 13:16:57.860195 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:57.859923 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" podUID="0b656366-2609-4928-83f1-18bc76cb1c48" containerName="istio-proxy" containerID="cri-o://363b77e3a283332a48e1ff503ed6a0ba7ed19bb3ede1e220df2dbf83b1a4cdc7" gracePeriod=30 Oct 30 13:16:57.872886 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:57.872853 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j"] Oct 30 13:16:57.873063 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:57.873045 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" podUID="0c9814ec-a81d-4763-b0cb-97e20bccd5f7" containerName="istio-proxy" containerID="cri-o://42672481b26f80e5b9d75232519d3cd2b75cc0abb0440bbc4b5282f30f69dafb" gracePeriod=30 Oct 30 13:16:58.378221 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:58.378185 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l"] Oct 30 13:16:58.378486 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:16:58.378458 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" podUID="af73e007-df5a-4dcb-9116-1a8764d29f7d" containerName="istio-proxy" containerID="cri-o://fb0246045bc84fc2c4a4db0357308ca170af2cdb439fda77193422354b58ddaa" gracePeriod=30 Oct 30 13:17:02.033412 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:02.033366 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc" Oct 30 13:17:02.046728 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:02.046696 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc" Oct 30 13:17:02.549755 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:02.549711 4931 scope.go:122] "RemoveContainer" containerID="b930c236469dfbe1bc7260c6f159fbd238fd94a04552478c72afddc686e69da8" Oct 30 13:17:02.560738 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:02.560716 4931 scope.go:122] "RemoveContainer" containerID="ac2b82f085dc7c61b3863c64fedeeb7cc0bc47953be3829ff19a7e871ddb2673" Oct 30 13:17:02.570599 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:02.570558 4931 scope.go:122] "RemoveContainer" containerID="e5f435be596b2cc97070d05e2b507610ef8b66b49e3352c680eb42841a60bbd8" Oct 30 13:17:02.716046 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:02.716011 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc" Oct 30 13:17:03.174679 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.174652 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" Oct 30 13:17:03.259496 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.259462 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b656366-2609-4928-83f1-18bc76cb1c48-kube-api-access-j2nfh\" (UniqueName: \"kubernetes.io/projected/0b656366-2609-4928-83f1-18bc76cb1c48-kube-api-access-j2nfh\") pod \"0b656366-2609-4928-83f1-18bc76cb1c48\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " Oct 30 13:17:03.259695 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.259586 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b656366-2609-4928-83f1-18bc76cb1c48-istio-token\" (UniqueName: \"kubernetes.io/projected/0b656366-2609-4928-83f1-18bc76cb1c48-istio-token\") pod \"0b656366-2609-4928-83f1-18bc76cb1c48\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " Oct 30 13:17:03.259695 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.259616 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b656366-2609-4928-83f1-18bc76cb1c48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b656366-2609-4928-83f1-18bc76cb1c48-istio-envoy\") pod \"0b656366-2609-4928-83f1-18bc76cb1c48\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " Oct 30 13:17:03.259815 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.259724 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b656366-2609-4928-83f1-18bc76cb1c48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b656366-2609-4928-83f1-18bc76cb1c48-istiod-ca-cert\") pod \"0b656366-2609-4928-83f1-18bc76cb1c48\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " Oct 30 13:17:03.259815 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.259766 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b656366-2609-4928-83f1-18bc76cb1c48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b656366-2609-4928-83f1-18bc76cb1c48-credential-socket\") pod \"0b656366-2609-4928-83f1-18bc76cb1c48\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " Oct 30 13:17:03.259815 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.259810 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b656366-2609-4928-83f1-18bc76cb1c48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b656366-2609-4928-83f1-18bc76cb1c48-workload-certs\") pod \"0b656366-2609-4928-83f1-18bc76cb1c48\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " Oct 30 13:17:03.259973 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.259919 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b656366-2609-4928-83f1-18bc76cb1c48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b656366-2609-4928-83f1-18bc76cb1c48-workload-socket\") pod \"0b656366-2609-4928-83f1-18bc76cb1c48\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " Oct 30 13:17:03.259973 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.259963 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b656366-2609-4928-83f1-18bc76cb1c48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b656366-2609-4928-83f1-18bc76cb1c48-istio-data\") pod \"0b656366-2609-4928-83f1-18bc76cb1c48\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " Oct 30 13:17:03.260081 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.260054 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b656366-2609-4928-83f1-18bc76cb1c48-credential-socket" pod "0b656366-2609-4928-83f1-18bc76cb1c48" (UID: "0b656366-2609-4928-83f1-18bc76cb1c48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:03.260081 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.260072 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b656366-2609-4928-83f1-18bc76cb1c48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b656366-2609-4928-83f1-18bc76cb1c48-istio-podinfo\") pod \"0b656366-2609-4928-83f1-18bc76cb1c48\" (UID: \"0b656366-2609-4928-83f1-18bc76cb1c48\") " Oct 30 13:17:03.260182 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.260100 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b656366-2609-4928-83f1-18bc76cb1c48-istiod-ca-cert" pod "0b656366-2609-4928-83f1-18bc76cb1c48" (UID: "0b656366-2609-4928-83f1-18bc76cb1c48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:17:03.260349 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.260292 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b656366-2609-4928-83f1-18bc76cb1c48-workload-socket" pod "0b656366-2609-4928-83f1-18bc76cb1c48" (UID: "0b656366-2609-4928-83f1-18bc76cb1c48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:03.260452 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.260340 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b656366-2609-4928-83f1-18bc76cb1c48-istio-data" pod "0b656366-2609-4928-83f1-18bc76cb1c48" (UID: "0b656366-2609-4928-83f1-18bc76cb1c48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:03.260452 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.260392 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b656366-2609-4928-83f1-18bc76cb1c48-workload-certs" pod "0b656366-2609-4928-83f1-18bc76cb1c48" (UID: "0b656366-2609-4928-83f1-18bc76cb1c48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:03.260800 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.260778 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b656366-2609-4928-83f1-18bc76cb1c48-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:17:03.260866 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.260808 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b656366-2609-4928-83f1-18bc76cb1c48-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:17:03.260866 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.260826 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b656366-2609-4928-83f1-18bc76cb1c48-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:17:03.260866 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.260841 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b656366-2609-4928-83f1-18bc76cb1c48-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:17:03.260866 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.260854 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b656366-2609-4928-83f1-18bc76cb1c48-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:17:03.261837 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.261811 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b656366-2609-4928-83f1-18bc76cb1c48-kube-api-access-j2nfh" pod "0b656366-2609-4928-83f1-18bc76cb1c48" (UID: "0b656366-2609-4928-83f1-18bc76cb1c48"). InnerVolumeSpecName "kube-api-access-j2nfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:17:03.262353 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.262328 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b656366-2609-4928-83f1-18bc76cb1c48-istio-envoy" pod "0b656366-2609-4928-83f1-18bc76cb1c48" (UID: "0b656366-2609-4928-83f1-18bc76cb1c48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:03.262465 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.262414 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b656366-2609-4928-83f1-18bc76cb1c48-istio-token" pod "0b656366-2609-4928-83f1-18bc76cb1c48" (UID: "0b656366-2609-4928-83f1-18bc76cb1c48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:17:03.262530 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.262483 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b656366-2609-4928-83f1-18bc76cb1c48-istio-podinfo" pod "0b656366-2609-4928-83f1-18bc76cb1c48" (UID: "0b656366-2609-4928-83f1-18bc76cb1c48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:17:03.362832 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.362796 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b656366-2609-4928-83f1-18bc76cb1c48-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:17:03.362832 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.362827 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2nfh\" (UniqueName: \"kubernetes.io/projected/0b656366-2609-4928-83f1-18bc76cb1c48-kube-api-access-j2nfh\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:17:03.362832 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.362837 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b656366-2609-4928-83f1-18bc76cb1c48-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:17:03.363074 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.362846 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b656366-2609-4928-83f1-18bc76cb1c48-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:17:03.550341 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.550313 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" Oct 30 13:17:03.669940 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.669908 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" Oct 30 13:17:03.675970 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.675942 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-istio-data\") pod \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " Oct 30 13:17:03.676118 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.675988 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-workload-certs\") pod \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " Oct 30 13:17:03.676118 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.676040 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-istio-podinfo\") pod \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " Oct 30 13:17:03.676118 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.676070 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-credential-socket\") pod \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " Oct 30 13:17:03.676292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.676182 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-workload-socket\") pod \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " Oct 30 13:17:03.676292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.676234 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-istio-token\" (UniqueName: \"kubernetes.io/projected/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-istio-token\") pod \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " Oct 30 13:17:03.676292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.676256 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-istio-data" pod "0c9814ec-a81d-4763-b0cb-97e20bccd5f7" (UID: "0c9814ec-a81d-4763-b0cb-97e20bccd5f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:03.676452 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.676297 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-istio-envoy\") pod \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " Oct 30 13:17:03.676452 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.676301 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-workload-certs" pod "0c9814ec-a81d-4763-b0cb-97e20bccd5f7" (UID: "0c9814ec-a81d-4763-b0cb-97e20bccd5f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:03.676452 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.676340 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-istiod-ca-cert\") pod \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " Oct 30 13:17:03.676621 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.676450 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-workload-socket" pod "0c9814ec-a81d-4763-b0cb-97e20bccd5f7" (UID: "0c9814ec-a81d-4763-b0cb-97e20bccd5f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:03.676621 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.676454 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-kube-api-access-fppzm\" (UniqueName: \"kubernetes.io/projected/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-kube-api-access-fppzm\") pod \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\" (UID: \"0c9814ec-a81d-4763-b0cb-97e20bccd5f7\") " Oct 30 13:17:03.676621 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.676479 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-credential-socket" pod "0c9814ec-a81d-4763-b0cb-97e20bccd5f7" (UID: "0c9814ec-a81d-4763-b0cb-97e20bccd5f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:03.676863 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.676825 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-istiod-ca-cert" pod "0c9814ec-a81d-4763-b0cb-97e20bccd5f7" (UID: "0c9814ec-a81d-4763-b0cb-97e20bccd5f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:17:03.677357 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.677251 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:17:03.677473 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.677372 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:17:03.677473 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.677385 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:17:03.677473 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.677395 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:17:03.677473 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.677405 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:17:03.678718 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.678696 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-kube-api-access-fppzm" pod "0c9814ec-a81d-4763-b0cb-97e20bccd5f7" (UID: "0c9814ec-a81d-4763-b0cb-97e20bccd5f7"). InnerVolumeSpecName "kube-api-access-fppzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:17:03.678857 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.678843 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-istio-token" pod "0c9814ec-a81d-4763-b0cb-97e20bccd5f7" (UID: "0c9814ec-a81d-4763-b0cb-97e20bccd5f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:17:03.678981 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.678962 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-istio-podinfo" pod "0c9814ec-a81d-4763-b0cb-97e20bccd5f7" (UID: "0c9814ec-a81d-4763-b0cb-97e20bccd5f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:17:03.679022 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.678977 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-istio-envoy" pod "0c9814ec-a81d-4763-b0cb-97e20bccd5f7" (UID: "0c9814ec-a81d-4763-b0cb-97e20bccd5f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:03.721579 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.721055 4931 generic.go:356] "Generic (PLEG): container finished" podID="af73e007-df5a-4dcb-9116-1a8764d29f7d" containerID="fb0246045bc84fc2c4a4db0357308ca170af2cdb439fda77193422354b58ddaa" exitCode=0 Oct 30 13:17:03.721579 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.721236 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" event={"ID":"af73e007-df5a-4dcb-9116-1a8764d29f7d","Type":"ContainerDied","Data":"fb0246045bc84fc2c4a4db0357308ca170af2cdb439fda77193422354b58ddaa"} Oct 30 13:17:03.721579 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.721298 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" event={"ID":"af73e007-df5a-4dcb-9116-1a8764d29f7d","Type":"ContainerDied","Data":"0c4962d8d8772f7bc835059ba11d9d6b971b0ee0b0e6afb708482fdf513d3fc5"} Oct 30 13:17:03.721579 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.721324 4931 scope.go:122] "RemoveContainer" containerID="fb0246045bc84fc2c4a4db0357308ca170af2cdb439fda77193422354b58ddaa" Oct 30 13:17:03.721579 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.721465 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l" Oct 30 13:17:03.724986 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.724955 4931 generic.go:356] "Generic (PLEG): container finished" podID="0b656366-2609-4928-83f1-18bc76cb1c48" containerID="363b77e3a283332a48e1ff503ed6a0ba7ed19bb3ede1e220df2dbf83b1a4cdc7" exitCode=0 Oct 30 13:17:03.725149 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.725126 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" event={"ID":"0b656366-2609-4928-83f1-18bc76cb1c48","Type":"ContainerDied","Data":"363b77e3a283332a48e1ff503ed6a0ba7ed19bb3ede1e220df2dbf83b1a4cdc7"} Oct 30 13:17:03.725224 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.725179 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" event={"ID":"0b656366-2609-4928-83f1-18bc76cb1c48","Type":"ContainerDied","Data":"2b895f6308c638466fd610fcd44fc11c0cd6a9566999e8156cee882aebea5a8a"} Oct 30 13:17:03.726212 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.725325 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz" Oct 30 13:17:03.727681 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.727656 4931 generic.go:356] "Generic (PLEG): container finished" podID="0c9814ec-a81d-4763-b0cb-97e20bccd5f7" containerID="42672481b26f80e5b9d75232519d3cd2b75cc0abb0440bbc4b5282f30f69dafb" exitCode=0 Oct 30 13:17:03.727804 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.727720 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" Oct 30 13:17:03.727804 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.727785 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" event={"ID":"0c9814ec-a81d-4763-b0cb-97e20bccd5f7","Type":"ContainerDied","Data":"42672481b26f80e5b9d75232519d3cd2b75cc0abb0440bbc4b5282f30f69dafb"} Oct 30 13:17:03.727899 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.727856 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j" event={"ID":"0c9814ec-a81d-4763-b0cb-97e20bccd5f7","Type":"ContainerDied","Data":"a3957c8488dd67cfe511f929ecf84528d699996a8aa4aa074f51c0e3e689ef27"} Oct 30 13:17:03.741536 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.741509 4931 scope.go:122] "RemoveContainer" containerID="fb0246045bc84fc2c4a4db0357308ca170af2cdb439fda77193422354b58ddaa" Oct 30 13:17:03.745210 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:17:03.745175 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb0246045bc84fc2c4a4db0357308ca170af2cdb439fda77193422354b58ddaa\": container with ID starting with fb0246045bc84fc2c4a4db0357308ca170af2cdb439fda77193422354b58ddaa not found: ID does not exist" containerID="fb0246045bc84fc2c4a4db0357308ca170af2cdb439fda77193422354b58ddaa" Oct 30 13:17:03.745392 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.745218 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb0246045bc84fc2c4a4db0357308ca170af2cdb439fda77193422354b58ddaa"} err="failed to get container status \"fb0246045bc84fc2c4a4db0357308ca170af2cdb439fda77193422354b58ddaa\": rpc error: code = NotFound desc = could not find container \"fb0246045bc84fc2c4a4db0357308ca170af2cdb439fda77193422354b58ddaa\": container with ID starting with fb0246045bc84fc2c4a4db0357308ca170af2cdb439fda77193422354b58ddaa not found: ID does not exist" Oct 30 13:17:03.745392 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.745243 4931 scope.go:122] "RemoveContainer" containerID="363b77e3a283332a48e1ff503ed6a0ba7ed19bb3ede1e220df2dbf83b1a4cdc7" Oct 30 13:17:03.757597 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.757573 4931 scope.go:122] "RemoveContainer" containerID="363b77e3a283332a48e1ff503ed6a0ba7ed19bb3ede1e220df2dbf83b1a4cdc7" Oct 30 13:17:03.757985 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:17:03.757959 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"363b77e3a283332a48e1ff503ed6a0ba7ed19bb3ede1e220df2dbf83b1a4cdc7\": container with ID starting with 363b77e3a283332a48e1ff503ed6a0ba7ed19bb3ede1e220df2dbf83b1a4cdc7 not found: ID does not exist" containerID="363b77e3a283332a48e1ff503ed6a0ba7ed19bb3ede1e220df2dbf83b1a4cdc7" Oct 30 13:17:03.758068 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.757995 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"363b77e3a283332a48e1ff503ed6a0ba7ed19bb3ede1e220df2dbf83b1a4cdc7"} err="failed to get container status \"363b77e3a283332a48e1ff503ed6a0ba7ed19bb3ede1e220df2dbf83b1a4cdc7\": rpc error: code = NotFound desc = could not find container \"363b77e3a283332a48e1ff503ed6a0ba7ed19bb3ede1e220df2dbf83b1a4cdc7\": container with ID starting with 363b77e3a283332a48e1ff503ed6a0ba7ed19bb3ede1e220df2dbf83b1a4cdc7 not found: ID does not exist" Oct 30 13:17:03.758068 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.758020 4931 scope.go:122] "RemoveContainer" containerID="42672481b26f80e5b9d75232519d3cd2b75cc0abb0440bbc4b5282f30f69dafb" Oct 30 13:17:03.768515 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.768498 4931 scope.go:122] "RemoveContainer" containerID="42672481b26f80e5b9d75232519d3cd2b75cc0abb0440bbc4b5282f30f69dafb" Oct 30 13:17:03.768777 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:17:03.768759 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42672481b26f80e5b9d75232519d3cd2b75cc0abb0440bbc4b5282f30f69dafb\": container with ID starting with 42672481b26f80e5b9d75232519d3cd2b75cc0abb0440bbc4b5282f30f69dafb not found: ID does not exist" containerID="42672481b26f80e5b9d75232519d3cd2b75cc0abb0440bbc4b5282f30f69dafb" Oct 30 13:17:03.768839 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.768784 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42672481b26f80e5b9d75232519d3cd2b75cc0abb0440bbc4b5282f30f69dafb"} err="failed to get container status \"42672481b26f80e5b9d75232519d3cd2b75cc0abb0440bbc4b5282f30f69dafb\": rpc error: code = NotFound desc = could not find container \"42672481b26f80e5b9d75232519d3cd2b75cc0abb0440bbc4b5282f30f69dafb\": container with ID starting with 42672481b26f80e5b9d75232519d3cd2b75cc0abb0440bbc4b5282f30f69dafb not found: ID does not exist" Oct 30 13:17:03.778339 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.778316 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af73e007-df5a-4dcb-9116-1a8764d29f7d-istio-token\" (UniqueName: \"kubernetes.io/projected/af73e007-df5a-4dcb-9116-1a8764d29f7d-istio-token\") pod \"af73e007-df5a-4dcb-9116-1a8764d29f7d\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " Oct 30 13:17:03.778443 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.778377 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af73e007-df5a-4dcb-9116-1a8764d29f7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af73e007-df5a-4dcb-9116-1a8764d29f7d-istio-envoy\") pod \"af73e007-df5a-4dcb-9116-1a8764d29f7d\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " Oct 30 13:17:03.778443 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.778416 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af73e007-df5a-4dcb-9116-1a8764d29f7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af73e007-df5a-4dcb-9116-1a8764d29f7d-istiod-ca-cert\") pod \"af73e007-df5a-4dcb-9116-1a8764d29f7d\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " Oct 30 13:17:03.778668 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.778644 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af73e007-df5a-4dcb-9116-1a8764d29f7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af73e007-df5a-4dcb-9116-1a8764d29f7d-istio-podinfo\") pod \"af73e007-df5a-4dcb-9116-1a8764d29f7d\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " Oct 30 13:17:03.778834 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.778715 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af73e007-df5a-4dcb-9116-1a8764d29f7d-kube-api-access-dwkdd\" (UniqueName: \"kubernetes.io/projected/af73e007-df5a-4dcb-9116-1a8764d29f7d-kube-api-access-dwkdd\") pod \"af73e007-df5a-4dcb-9116-1a8764d29f7d\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " Oct 30 13:17:03.778834 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.778761 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af73e007-df5a-4dcb-9116-1a8764d29f7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af73e007-df5a-4dcb-9116-1a8764d29f7d-istio-data\") pod \"af73e007-df5a-4dcb-9116-1a8764d29f7d\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " Oct 30 13:17:03.783382 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.778888 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af73e007-df5a-4dcb-9116-1a8764d29f7d-istiod-ca-cert" pod "af73e007-df5a-4dcb-9116-1a8764d29f7d" (UID: "af73e007-df5a-4dcb-9116-1a8764d29f7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:17:03.783382 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.779315 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af73e007-df5a-4dcb-9116-1a8764d29f7d-istio-data" pod "af73e007-df5a-4dcb-9116-1a8764d29f7d" (UID: "af73e007-df5a-4dcb-9116-1a8764d29f7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:03.783382 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.779522 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af73e007-df5a-4dcb-9116-1a8764d29f7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af73e007-df5a-4dcb-9116-1a8764d29f7d-workload-certs\") pod \"af73e007-df5a-4dcb-9116-1a8764d29f7d\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " Oct 30 13:17:03.783382 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.779564 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af73e007-df5a-4dcb-9116-1a8764d29f7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af73e007-df5a-4dcb-9116-1a8764d29f7d-credential-socket\") pod \"af73e007-df5a-4dcb-9116-1a8764d29f7d\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " Oct 30 13:17:03.783382 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.779717 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af73e007-df5a-4dcb-9116-1a8764d29f7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af73e007-df5a-4dcb-9116-1a8764d29f7d-workload-socket\") pod \"af73e007-df5a-4dcb-9116-1a8764d29f7d\" (UID: \"af73e007-df5a-4dcb-9116-1a8764d29f7d\") " Oct 30 13:17:03.783382 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.779966 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af73e007-df5a-4dcb-9116-1a8764d29f7d-workload-certs" pod "af73e007-df5a-4dcb-9116-1a8764d29f7d" (UID: "af73e007-df5a-4dcb-9116-1a8764d29f7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:03.783382 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.780564 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:17:03.783382 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.780585 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:17:03.783382 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.780599 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af73e007-df5a-4dcb-9116-1a8764d29f7d-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:17:03.783382 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.780614 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af73e007-df5a-4dcb-9116-1a8764d29f7d-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:17:03.783382 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.780628 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fppzm\" (UniqueName: \"kubernetes.io/projected/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-kube-api-access-fppzm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:17:03.783382 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.780641 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af73e007-df5a-4dcb-9116-1a8764d29f7d-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:17:03.783382 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.780657 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9814ec-a81d-4763-b0cb-97e20bccd5f7-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:17:03.783382 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.780878 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af73e007-df5a-4dcb-9116-1a8764d29f7d-workload-socket" pod "af73e007-df5a-4dcb-9116-1a8764d29f7d" (UID: "af73e007-df5a-4dcb-9116-1a8764d29f7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:03.783382 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.781175 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af73e007-df5a-4dcb-9116-1a8764d29f7d-istio-envoy" pod "af73e007-df5a-4dcb-9116-1a8764d29f7d" (UID: "af73e007-df5a-4dcb-9116-1a8764d29f7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:03.783382 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.781910 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af73e007-df5a-4dcb-9116-1a8764d29f7d-istio-podinfo" pod "af73e007-df5a-4dcb-9116-1a8764d29f7d" (UID: "af73e007-df5a-4dcb-9116-1a8764d29f7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:17:03.783382 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.782250 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af73e007-df5a-4dcb-9116-1a8764d29f7d-istio-token" pod "af73e007-df5a-4dcb-9116-1a8764d29f7d" (UID: "af73e007-df5a-4dcb-9116-1a8764d29f7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:17:03.783382 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.782664 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af73e007-df5a-4dcb-9116-1a8764d29f7d-credential-socket" pod "af73e007-df5a-4dcb-9116-1a8764d29f7d" (UID: "af73e007-df5a-4dcb-9116-1a8764d29f7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:03.783382 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.782978 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af73e007-df5a-4dcb-9116-1a8764d29f7d-kube-api-access-dwkdd" pod "af73e007-df5a-4dcb-9116-1a8764d29f7d" (UID: "af73e007-df5a-4dcb-9116-1a8764d29f7d"). InnerVolumeSpecName "kube-api-access-dwkdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:17:03.785879 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.785851 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j"] Oct 30 13:17:03.789624 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.789601 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j"] Oct 30 13:17:03.823929 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.823892 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz"] Oct 30 13:17:03.835894 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.835856 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz"] Oct 30 13:17:03.882010 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.881976 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af73e007-df5a-4dcb-9116-1a8764d29f7d-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:17:03.882010 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.882006 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af73e007-df5a-4dcb-9116-1a8764d29f7d-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:17:03.882010 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.882018 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af73e007-df5a-4dcb-9116-1a8764d29f7d-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:17:03.882254 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.882027 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af73e007-df5a-4dcb-9116-1a8764d29f7d-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:17:03.882254 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.882035 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwkdd\" (UniqueName: \"kubernetes.io/projected/af73e007-df5a-4dcb-9116-1a8764d29f7d-kube-api-access-dwkdd\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:17:03.882254 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:03.882046 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af73e007-df5a-4dcb-9116-1a8764d29f7d-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:17:04.072178 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:17:04.072140 4931 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf73e007_df5a_4dcb_9116_1a8764d29f7d.slice\": RecentStats: unable to find data in memory cache]" Oct 30 13:17:04.074731 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:04.074691 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l"] Oct 30 13:17:04.078735 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:04.078710 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l"] Oct 30 13:17:05.514533 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:05.514497 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b656366-2609-4928-83f1-18bc76cb1c48" path="/var/lib/kubelet/pods/0b656366-2609-4928-83f1-18bc76cb1c48/volumes" Oct 30 13:17:05.514996 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:05.514981 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c9814ec-a81d-4763-b0cb-97e20bccd5f7" path="/var/lib/kubelet/pods/0c9814ec-a81d-4763-b0cb-97e20bccd5f7/volumes" Oct 30 13:17:05.515458 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:05.515443 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af73e007-df5a-4dcb-9116-1a8764d29f7d" path="/var/lib/kubelet/pods/af73e007-df5a-4dcb-9116-1a8764d29f7d/volumes" Oct 30 13:17:07.352795 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:17:07.352756 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:17:08.052354 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:08.052321 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58f9b7b5-xsk46"] Oct 30 13:17:08.064606 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:08.064571 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58f9b7b5-xsk46"] Oct 30 13:17:09.514885 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:09.514845 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d85da789-4fa7-41e5-a7f3-6b16031020fe" path="/var/lib/kubelet/pods/d85da789-4fa7-41e5-a7f3-6b16031020fe/volumes" Oct 30 13:17:15.895346 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:15.893104 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:17:27.302727 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.302684 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz"] Oct 30 13:17:27.303482 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.303461 4931 cpu_manager.go:414] "containerMap: removing container" podUID="af73e007-df5a-4dcb-9116-1a8764d29f7d" containerName="istio-proxy" Oct 30 13:17:27.303482 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.303482 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af73e007-df5a-4dcb-9116-1a8764d29f7d" containerName="istio-proxy" Oct 30 13:17:27.303657 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.303512 4931 cpu_manager.go:414] "containerMap: removing container" podUID="0b656366-2609-4928-83f1-18bc76cb1c48" containerName="istio-proxy" Oct 30 13:17:27.303657 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.303520 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b656366-2609-4928-83f1-18bc76cb1c48" containerName="istio-proxy" Oct 30 13:17:27.303657 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.303559 4931 cpu_manager.go:414] "containerMap: removing container" podUID="0c9814ec-a81d-4763-b0cb-97e20bccd5f7" containerName="istio-proxy" Oct 30 13:17:27.303657 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.303567 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c9814ec-a81d-4763-b0cb-97e20bccd5f7" containerName="istio-proxy" Oct 30 13:17:27.303849 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.303734 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c9814ec-a81d-4763-b0cb-97e20bccd5f7" containerName="istio-proxy" Oct 30 13:17:27.303849 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.303756 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="af73e007-df5a-4dcb-9116-1a8764d29f7d" containerName="istio-proxy" Oct 30 13:17:27.303849 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.303769 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b656366-2609-4928-83f1-18bc76cb1c48" containerName="istio-proxy" Oct 30 13:17:27.307578 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.307551 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz" Oct 30 13:17:27.328893 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.328866 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz"] Oct 30 13:17:27.336827 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.336802 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4a8ee1\"/\"openshift-service-ca.crt\"" Oct 30 13:17:27.336977 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.336889 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4a8ee1\"/\"kube-root-ca.crt\"" Oct 30 13:17:27.336977 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.336916 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4a8ee1\"/\"default-dockercfg-z5kk4\"" Oct 30 13:17:27.336977 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.336959 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4a8ee1\"/\"test-model-4a8ee1-kserve-self-signed-certs\"" Oct 30 13:17:27.465531 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.465491 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8nn6\" (UniqueName: \"kubernetes.io/projected/5c77e909-e679-4e88-8569-b406b4582f75-kube-api-access-s8nn6\") pod \"test-model-4a8ee1-kserve-766c76b76c-wldrz\" (UID: \"5c77e909-e679-4e88-8569-b406b4582f75\") " pod="ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz" Oct 30 13:17:27.465720 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.465548 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c77e909-e679-4e88-8569-b406b4582f75-dshm\") pod \"test-model-4a8ee1-kserve-766c76b76c-wldrz\" (UID: \"5c77e909-e679-4e88-8569-b406b4582f75\") " pod="ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz" Oct 30 13:17:27.465720 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.465643 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c77e909-e679-4e88-8569-b406b4582f75-tmp-dir\") pod \"test-model-4a8ee1-kserve-766c76b76c-wldrz\" (UID: \"5c77e909-e679-4e88-8569-b406b4582f75\") " pod="ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz" Oct 30 13:17:27.465720 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.465710 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c77e909-e679-4e88-8569-b406b4582f75-tls-certs\") pod \"test-model-4a8ee1-kserve-766c76b76c-wldrz\" (UID: \"5c77e909-e679-4e88-8569-b406b4582f75\") " pod="ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz" Oct 30 13:17:27.465930 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.465840 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c77e909-e679-4e88-8569-b406b4582f75-model-cache\") pod \"test-model-4a8ee1-kserve-766c76b76c-wldrz\" (UID: \"5c77e909-e679-4e88-8569-b406b4582f75\") " pod="ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz" Oct 30 13:17:27.466083 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.466063 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c77e909-e679-4e88-8569-b406b4582f75-home\") pod \"test-model-4a8ee1-kserve-766c76b76c-wldrz\" (UID: \"5c77e909-e679-4e88-8569-b406b4582f75\") " pod="ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz" Oct 30 13:17:27.575261 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.575156 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c77e909-e679-4e88-8569-b406b4582f75-tls-certs\") pod \"test-model-4a8ee1-kserve-766c76b76c-wldrz\" (UID: \"5c77e909-e679-4e88-8569-b406b4582f75\") " pod="ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz" Oct 30 13:17:27.575261 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.575252 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c77e909-e679-4e88-8569-b406b4582f75-model-cache\") pod \"test-model-4a8ee1-kserve-766c76b76c-wldrz\" (UID: \"5c77e909-e679-4e88-8569-b406b4582f75\") " pod="ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz" Oct 30 13:17:27.575523 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.575452 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c77e909-e679-4e88-8569-b406b4582f75-home\") pod \"test-model-4a8ee1-kserve-766c76b76c-wldrz\" (UID: \"5c77e909-e679-4e88-8569-b406b4582f75\") " pod="ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz" Oct 30 13:17:27.575587 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.575551 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8nn6\" (UniqueName: \"kubernetes.io/projected/5c77e909-e679-4e88-8569-b406b4582f75-kube-api-access-s8nn6\") pod \"test-model-4a8ee1-kserve-766c76b76c-wldrz\" (UID: \"5c77e909-e679-4e88-8569-b406b4582f75\") " pod="ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz" Oct 30 13:17:27.575641 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.575586 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c77e909-e679-4e88-8569-b406b4582f75-dshm\") pod \"test-model-4a8ee1-kserve-766c76b76c-wldrz\" (UID: \"5c77e909-e679-4e88-8569-b406b4582f75\") " pod="ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz" Oct 30 13:17:27.575641 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.575625 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c77e909-e679-4e88-8569-b406b4582f75-tmp-dir\") pod \"test-model-4a8ee1-kserve-766c76b76c-wldrz\" (UID: \"5c77e909-e679-4e88-8569-b406b4582f75\") " pod="ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz" Oct 30 13:17:27.575992 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.575964 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c77e909-e679-4e88-8569-b406b4582f75-model-cache\") pod \"test-model-4a8ee1-kserve-766c76b76c-wldrz\" (UID: \"5c77e909-e679-4e88-8569-b406b4582f75\") " pod="ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz" Oct 30 13:17:27.576098 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.576026 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c77e909-e679-4e88-8569-b406b4582f75-tmp-dir\") pod \"test-model-4a8ee1-kserve-766c76b76c-wldrz\" (UID: \"5c77e909-e679-4e88-8569-b406b4582f75\") " pod="ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz" Oct 30 13:17:27.576706 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.576679 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c77e909-e679-4e88-8569-b406b4582f75-home\") pod \"test-model-4a8ee1-kserve-766c76b76c-wldrz\" (UID: \"5c77e909-e679-4e88-8569-b406b4582f75\") " pod="ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz" Oct 30 13:17:27.593125 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.590176 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c77e909-e679-4e88-8569-b406b4582f75-tls-certs\") pod \"test-model-4a8ee1-kserve-766c76b76c-wldrz\" (UID: \"5c77e909-e679-4e88-8569-b406b4582f75\") " pod="ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz" Oct 30 13:17:27.600315 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.599650 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c77e909-e679-4e88-8569-b406b4582f75-dshm\") pod \"test-model-4a8ee1-kserve-766c76b76c-wldrz\" (UID: \"5c77e909-e679-4e88-8569-b406b4582f75\") " pod="ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz" Oct 30 13:17:27.622303 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.614073 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8nn6\" (UniqueName: \"kubernetes.io/projected/5c77e909-e679-4e88-8569-b406b4582f75-kube-api-access-s8nn6\") pod \"test-model-4a8ee1-kserve-766c76b76c-wldrz\" (UID: \"5c77e909-e679-4e88-8569-b406b4582f75\") " pod="ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz" Oct 30 13:17:27.624161 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.623892 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz" Oct 30 13:17:27.990248 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:27.990210 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz"] Oct 30 13:17:27.990552 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:17:27.990524 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c77e909_e679_4e88_8569_b406b4582f75.slice/crio-39177a3c3a0987ce2deb015c94744c885bd6b6fc01778c4ab2b498a485166268 WatchSource:0}: Error finding container 39177a3c3a0987ce2deb015c94744c885bd6b6fc01778c4ab2b498a485166268: Status 404 returned error can't find the container with id 39177a3c3a0987ce2deb015c94744c885bd6b6fc01778c4ab2b498a485166268 Oct 30 13:17:28.935812 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:28.935774 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz" event={"ID":"5c77e909-e679-4e88-8569-b406b4582f75","Type":"ContainerStarted","Data":"0ead8176769b627ba26636c9327e0960be0f784ad424b76cb4776d9e707fe2cd"} Oct 30 13:17:28.935812 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:28.935820 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz" event={"ID":"5c77e909-e679-4e88-8569-b406b4582f75","Type":"ContainerStarted","Data":"39177a3c3a0987ce2deb015c94744c885bd6b6fc01778c4ab2b498a485166268"} Oct 30 13:17:28.984486 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:28.984426 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz" podStartSLOduration=1.984411677 podStartE2EDuration="1.984411677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:28.980991089 +0000 UTC m=+1947.584489539" watchObservedRunningTime="2026-10-30 13:17:28.984411677 +0000 UTC m=+1947.587910124" Oct 30 13:17:29.624918 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:29.624864 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz" Oct 30 13:17:37.354405 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:17:37.354358 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:17:37.624818 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:37.624706 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz" Oct 30 13:17:37.637764 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:37.637729 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz" Oct 30 13:17:38.013386 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:38.013351 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz" Oct 30 13:17:49.275050 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:49.275014 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j"] Oct 30 13:17:49.280757 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:49.280735 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j" Oct 30 13:17:49.299170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:49.299142 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-cd2d2e-serving-cert\"" Oct 30 13:17:49.314221 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:49.314178 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j"] Oct 30 13:17:49.416394 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:49.416352 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx8z6\" (UniqueName: \"kubernetes.io/projected/59884998-e47d-448b-bf1d-9911a41b70c1-kube-api-access-qx8z6\") pod \"maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j\" (UID: \"59884998-e47d-448b-bf1d-9911a41b70c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j" Oct 30 13:17:49.416394 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:49.416399 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59884998-e47d-448b-bf1d-9911a41b70c1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j\" (UID: \"59884998-e47d-448b-bf1d-9911a41b70c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j" Oct 30 13:17:49.416611 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:49.416549 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59884998-e47d-448b-bf1d-9911a41b70c1-metrics-certs\") pod \"maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j\" (UID: \"59884998-e47d-448b-bf1d-9911a41b70c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j" Oct 30 13:17:49.416677 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:49.416659 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59884998-e47d-448b-bf1d-9911a41b70c1-maas-api-tls\") pod \"maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j\" (UID: \"59884998-e47d-448b-bf1d-9911a41b70c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j" Oct 30 13:17:49.518583 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:49.518534 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx8z6\" (UniqueName: \"kubernetes.io/projected/59884998-e47d-448b-bf1d-9911a41b70c1-kube-api-access-qx8z6\") pod \"maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j\" (UID: \"59884998-e47d-448b-bf1d-9911a41b70c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j" Oct 30 13:17:49.518583 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:49.518603 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59884998-e47d-448b-bf1d-9911a41b70c1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j\" (UID: \"59884998-e47d-448b-bf1d-9911a41b70c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j" Oct 30 13:17:49.518907 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:49.518727 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59884998-e47d-448b-bf1d-9911a41b70c1-metrics-certs\") pod \"maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j\" (UID: \"59884998-e47d-448b-bf1d-9911a41b70c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j" Oct 30 13:17:49.518907 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:49.518863 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59884998-e47d-448b-bf1d-9911a41b70c1-maas-api-tls\") pod \"maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j\" (UID: \"59884998-e47d-448b-bf1d-9911a41b70c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j" Oct 30 13:17:49.519542 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:49.519512 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59884998-e47d-448b-bf1d-9911a41b70c1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j\" (UID: \"59884998-e47d-448b-bf1d-9911a41b70c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j" Oct 30 13:17:49.521642 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:49.521619 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59884998-e47d-448b-bf1d-9911a41b70c1-maas-api-tls\") pod \"maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j\" (UID: \"59884998-e47d-448b-bf1d-9911a41b70c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j" Oct 30 13:17:49.521730 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:49.521689 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59884998-e47d-448b-bf1d-9911a41b70c1-metrics-certs\") pod \"maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j\" (UID: \"59884998-e47d-448b-bf1d-9911a41b70c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j" Oct 30 13:17:49.544817 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:49.544746 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx8z6\" (UniqueName: \"kubernetes.io/projected/59884998-e47d-448b-bf1d-9911a41b70c1-kube-api-access-qx8z6\") pod \"maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j\" (UID: \"59884998-e47d-448b-bf1d-9911a41b70c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j" Oct 30 13:17:49.601982 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:49.601948 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j" Oct 30 13:17:49.914720 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:49.914689 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j"] Oct 30 13:17:49.915322 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:17:49.915291 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59884998_e47d_448b_bf1d_9911a41b70c1.slice/crio-36183a5ff25f8f9381f6bbb49ba8dfe0d453c8ed7f732d4418ac4fa3167c0113 WatchSource:0}: Error finding container 36183a5ff25f8f9381f6bbb49ba8dfe0d453c8ed7f732d4418ac4fa3167c0113: Status 404 returned error can't find the container with id 36183a5ff25f8f9381f6bbb49ba8dfe0d453c8ed7f732d4418ac4fa3167c0113 Oct 30 13:17:50.099837 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:50.099788 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j" event={"ID":"59884998-e47d-448b-bf1d-9911a41b70c1","Type":"ContainerStarted","Data":"36183a5ff25f8f9381f6bbb49ba8dfe0d453c8ed7f732d4418ac4fa3167c0113"} Oct 30 13:17:51.108093 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:51.108053 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j" event={"ID":"59884998-e47d-448b-bf1d-9911a41b70c1","Type":"ContainerStarted","Data":"12fd1bb96ef84c5f4200ac70397f15b079e21c8d21dd5458f5e3776ec93a68d9"} Oct 30 13:17:51.108527 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:51.108182 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j" Oct 30 13:17:51.159739 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:51.159688 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j" podStartSLOduration=1.740250544 podStartE2EDuration="2.159674963s" totalImagesPullingTime="419.424419ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:51.157472812 +0000 UTC m=+1969.760971257" watchObservedRunningTime="2026-10-30 13:17:51.159674963 +0000 UTC m=+1969.763173409" Oct 30 13:17:51.201105 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:51.201065 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:17:56.474059 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:56.474020 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2"] Oct 30 13:17:56.479449 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:56.479421 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2" Oct 30 13:17:56.499511 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:56.499428 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6f9071\"/\"openshift-service-ca.crt\"" Oct 30 13:17:56.513091 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:56.513065 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6f9071\"/\"kube-root-ca.crt\"" Oct 30 13:17:56.513241 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:56.513075 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6f9071\"/\"default-dockercfg-mvfdb\"" Oct 30 13:17:56.513241 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:56.513098 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6f9071\"/\"test-model-6f9071-kserve-self-signed-certs\"" Oct 30 13:17:56.513619 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:56.513599 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-dshm\") pod \"test-model-6f9071-kserve-849958c557-x72j2\" (UID: \"3eb1597f-ffc6-46f2-a6fb-34e000c3ef83\") " pod="ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2" Oct 30 13:17:56.513674 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:56.513662 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-tmp-dir\") pod \"test-model-6f9071-kserve-849958c557-x72j2\" (UID: \"3eb1597f-ffc6-46f2-a6fb-34e000c3ef83\") " pod="ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2" Oct 30 13:17:56.513844 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:56.513821 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-model-cache\") pod \"test-model-6f9071-kserve-849958c557-x72j2\" (UID: \"3eb1597f-ffc6-46f2-a6fb-34e000c3ef83\") " pod="ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2" Oct 30 13:17:56.513935 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:56.513861 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-home\") pod \"test-model-6f9071-kserve-849958c557-x72j2\" (UID: \"3eb1597f-ffc6-46f2-a6fb-34e000c3ef83\") " pod="ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2" Oct 30 13:17:56.514003 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:56.513976 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fctwf\" (UniqueName: \"kubernetes.io/projected/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-kube-api-access-fctwf\") pod \"test-model-6f9071-kserve-849958c557-x72j2\" (UID: \"3eb1597f-ffc6-46f2-a6fb-34e000c3ef83\") " pod="ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2" Oct 30 13:17:56.514120 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:56.514099 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-tls-certs\") pod \"test-model-6f9071-kserve-849958c557-x72j2\" (UID: \"3eb1597f-ffc6-46f2-a6fb-34e000c3ef83\") " pod="ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2" Oct 30 13:17:56.520369 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:56.520347 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2"] Oct 30 13:17:56.615561 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:56.615522 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fctwf\" (UniqueName: \"kubernetes.io/projected/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-kube-api-access-fctwf\") pod \"test-model-6f9071-kserve-849958c557-x72j2\" (UID: \"3eb1597f-ffc6-46f2-a6fb-34e000c3ef83\") " pod="ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2" Oct 30 13:17:56.615749 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:56.615626 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-tls-certs\") pod \"test-model-6f9071-kserve-849958c557-x72j2\" (UID: \"3eb1597f-ffc6-46f2-a6fb-34e000c3ef83\") " pod="ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2" Oct 30 13:17:56.615798 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:56.615752 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-dshm\") pod \"test-model-6f9071-kserve-849958c557-x72j2\" (UID: \"3eb1597f-ffc6-46f2-a6fb-34e000c3ef83\") " pod="ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2" Oct 30 13:17:56.615855 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:56.615839 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-tmp-dir\") pod \"test-model-6f9071-kserve-849958c557-x72j2\" (UID: \"3eb1597f-ffc6-46f2-a6fb-34e000c3ef83\") " pod="ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2" Oct 30 13:17:56.615977 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:56.615960 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-model-cache\") pod \"test-model-6f9071-kserve-849958c557-x72j2\" (UID: \"3eb1597f-ffc6-46f2-a6fb-34e000c3ef83\") " pod="ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2" Oct 30 13:17:56.616067 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:56.615994 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-home\") pod \"test-model-6f9071-kserve-849958c557-x72j2\" (UID: \"3eb1597f-ffc6-46f2-a6fb-34e000c3ef83\") " pod="ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2" Oct 30 13:17:56.616317 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:56.616255 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-tmp-dir\") pod \"test-model-6f9071-kserve-849958c557-x72j2\" (UID: \"3eb1597f-ffc6-46f2-a6fb-34e000c3ef83\") " pod="ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2" Oct 30 13:17:56.616429 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:56.616380 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-model-cache\") pod \"test-model-6f9071-kserve-849958c557-x72j2\" (UID: \"3eb1597f-ffc6-46f2-a6fb-34e000c3ef83\") " pod="ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2" Oct 30 13:17:56.616492 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:56.616461 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-home\") pod \"test-model-6f9071-kserve-849958c557-x72j2\" (UID: \"3eb1597f-ffc6-46f2-a6fb-34e000c3ef83\") " pod="ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2" Oct 30 13:17:56.618615 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:56.618589 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-dshm\") pod \"test-model-6f9071-kserve-849958c557-x72j2\" (UID: \"3eb1597f-ffc6-46f2-a6fb-34e000c3ef83\") " pod="ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2" Oct 30 13:17:56.618811 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:56.618793 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-tls-certs\") pod \"test-model-6f9071-kserve-849958c557-x72j2\" (UID: \"3eb1597f-ffc6-46f2-a6fb-34e000c3ef83\") " pod="ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2" Oct 30 13:17:56.646830 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:56.646785 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fctwf\" (UniqueName: \"kubernetes.io/projected/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-kube-api-access-fctwf\") pod \"test-model-6f9071-kserve-849958c557-x72j2\" (UID: \"3eb1597f-ffc6-46f2-a6fb-34e000c3ef83\") " pod="ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2" Oct 30 13:17:56.795951 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:56.795834 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2" Oct 30 13:17:57.121610 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:57.121526 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j" Oct 30 13:17:57.227847 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:17:57.227799 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3eb1597f_ffc6_46f2_a6fb_34e000c3ef83.slice/crio-d3e9a705446ee1ee9ead89f72ace845cc0796572843bfe5be973a79a98e9ed94 WatchSource:0}: Error finding container d3e9a705446ee1ee9ead89f72ace845cc0796572843bfe5be973a79a98e9ed94: Status 404 returned error can't find the container with id d3e9a705446ee1ee9ead89f72ace845cc0796572843bfe5be973a79a98e9ed94 Oct 30 13:17:57.235246 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:57.231616 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2"] Oct 30 13:17:58.168542 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:58.168501 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2" event={"ID":"3eb1597f-ffc6-46f2-a6fb-34e000c3ef83","Type":"ContainerStarted","Data":"d4d850c54d777523ace5cc98af21d3645588353a094b1cf2b73e765a8116f907"} Oct 30 13:17:58.169001 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:58.168550 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2" event={"ID":"3eb1597f-ffc6-46f2-a6fb-34e000c3ef83","Type":"ContainerStarted","Data":"d3e9a705446ee1ee9ead89f72ace845cc0796572843bfe5be973a79a98e9ed94"} Oct 30 13:17:58.224280 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:58.224216 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2" podStartSLOduration=2.224199347 podStartE2EDuration="2.224199347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:58.220289002 +0000 UTC m=+1976.823787442" watchObservedRunningTime="2026-10-30 13:17:58.224199347 +0000 UTC m=+1976.827697794" Oct 30 13:17:58.797142 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:17:58.797106 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2" Oct 30 13:18:01.459933 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:01.459896 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp"] Oct 30 13:18:01.464377 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:01.464355 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp" Oct 30 13:18:01.496176 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:01.496148 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-618c72-serving-cert\"" Oct 30 13:18:01.501995 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:01.501968 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp"] Oct 30 13:18:01.602005 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:01.601962 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6b7f53a-3de6-4548-b270-48cb0a46030e-metrics-certs\") pod \"maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp\" (UID: \"e6b7f53a-3de6-4548-b270-48cb0a46030e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp" Oct 30 13:18:01.602429 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:01.602396 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6b7f53a-3de6-4548-b270-48cb0a46030e-maas-api-tls\") pod \"maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp\" (UID: \"e6b7f53a-3de6-4548-b270-48cb0a46030e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp" Oct 30 13:18:01.602548 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:01.602450 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6b7f53a-3de6-4548-b270-48cb0a46030e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp\" (UID: \"e6b7f53a-3de6-4548-b270-48cb0a46030e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp" Oct 30 13:18:01.602548 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:01.602530 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkn47\" (UniqueName: \"kubernetes.io/projected/e6b7f53a-3de6-4548-b270-48cb0a46030e-kube-api-access-xkn47\") pod \"maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp\" (UID: \"e6b7f53a-3de6-4548-b270-48cb0a46030e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp" Oct 30 13:18:01.703595 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:01.703561 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6b7f53a-3de6-4548-b270-48cb0a46030e-maas-api-tls\") pod \"maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp\" (UID: \"e6b7f53a-3de6-4548-b270-48cb0a46030e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp" Oct 30 13:18:01.703595 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:01.703599 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6b7f53a-3de6-4548-b270-48cb0a46030e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp\" (UID: \"e6b7f53a-3de6-4548-b270-48cb0a46030e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp" Oct 30 13:18:01.703806 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:01.703654 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkn47\" (UniqueName: \"kubernetes.io/projected/e6b7f53a-3de6-4548-b270-48cb0a46030e-kube-api-access-xkn47\") pod \"maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp\" (UID: \"e6b7f53a-3de6-4548-b270-48cb0a46030e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp" Oct 30 13:18:01.703806 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:01.703721 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6b7f53a-3de6-4548-b270-48cb0a46030e-metrics-certs\") pod \"maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp\" (UID: \"e6b7f53a-3de6-4548-b270-48cb0a46030e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp" Oct 30 13:18:01.704397 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:01.704361 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6b7f53a-3de6-4548-b270-48cb0a46030e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp\" (UID: \"e6b7f53a-3de6-4548-b270-48cb0a46030e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp" Oct 30 13:18:01.706250 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:01.706226 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6b7f53a-3de6-4548-b270-48cb0a46030e-metrics-certs\") pod \"maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp\" (UID: \"e6b7f53a-3de6-4548-b270-48cb0a46030e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp" Oct 30 13:18:01.719043 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:01.718991 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-618c72-serving-cert\"" Oct 30 13:18:01.726139 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:01.726118 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6b7f53a-3de6-4548-b270-48cb0a46030e-maas-api-tls\") pod \"maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp\" (UID: \"e6b7f53a-3de6-4548-b270-48cb0a46030e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp" Oct 30 13:18:01.728356 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:01.728332 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkn47\" (UniqueName: \"kubernetes.io/projected/e6b7f53a-3de6-4548-b270-48cb0a46030e-kube-api-access-xkn47\") pod \"maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp\" (UID: \"e6b7f53a-3de6-4548-b270-48cb0a46030e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp" Oct 30 13:18:01.779117 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:01.779077 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp" Oct 30 13:18:02.115995 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:18:02.115951 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6b7f53a_3de6_4548_b270_48cb0a46030e.slice/crio-b340493e94a88aaf51fc2c397ff614c48fe83ae9a19e4e6cca86c4f317e5c315 WatchSource:0}: Error finding container b340493e94a88aaf51fc2c397ff614c48fe83ae9a19e4e6cca86c4f317e5c315: Status 404 returned error can't find the container with id b340493e94a88aaf51fc2c397ff614c48fe83ae9a19e4e6cca86c4f317e5c315 Oct 30 13:18:02.117230 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:02.117199 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp"] Oct 30 13:18:02.203330 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:02.203291 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp" event={"ID":"e6b7f53a-3de6-4548-b270-48cb0a46030e","Type":"ContainerStarted","Data":"b340493e94a88aaf51fc2c397ff614c48fe83ae9a19e4e6cca86c4f317e5c315"} Oct 30 13:18:02.624360 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:02.624320 4931 scope.go:122] "RemoveContainer" containerID="7ad486931063396f67048d748c0088b6fb03845f0e8b3f3416573523a613475b" Oct 30 13:18:03.212098 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:03.212058 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp" event={"ID":"e6b7f53a-3de6-4548-b270-48cb0a46030e","Type":"ContainerStarted","Data":"0f24429675f77e9fca8c767af98420305b85ab7bc468c4f1c80b2a604c81135d"} Oct 30 13:18:03.212333 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:03.212152 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp" Oct 30 13:18:03.260569 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:03.260517 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp" podStartSLOduration=1.9097036969999999 podStartE2EDuration="2.260503559s" totalImagesPullingTime="350.799862ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:03.257359446 +0000 UTC m=+1981.860857893" watchObservedRunningTime="2026-10-30 13:18:03.260503559 +0000 UTC m=+1981.864002023" Oct 30 13:18:06.796796 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:06.796756 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2" Oct 30 13:18:06.811411 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:06.811380 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2" Oct 30 13:18:07.256805 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:07.256771 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2" Oct 30 13:18:07.357365 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:18:07.357334 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:18:09.227089 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:09.226795 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp" Oct 30 13:18:13.270956 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:13.270921 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14e8a4-z4jfb"] Oct 30 13:18:13.283046 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:13.283016 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14e8a4-z4jfb"] Oct 30 13:18:13.514573 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:13.514539 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1" path="/var/lib/kubelet/pods/f7ce9e0a-5781-41a8-8c60-98fa9ecce6b1/volumes" Oct 30 13:18:32.433355 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:32.433317 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-11cfa6-775hp"] Oct 30 13:18:32.446555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:32.446518 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-11cfa6-775hp"] Oct 30 13:18:33.515134 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:33.515089 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eba97ae-a62f-4ae2-bc04-e28404c7dc21" path="/var/lib/kubelet/pods/4eba97ae-a62f-4ae2-bc04-e28404c7dc21/volumes" Oct 30 13:18:37.353139 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:18:37.353103 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:18:40.211058 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:40.211023 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf"] Oct 30 13:18:40.211677 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:40.211276 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" podUID="9ef8a128-1b22-4ef2-81ed-0a1b494393c3" containerName="istio-proxy" containerID="cri-o://528bea8ad0c34002c1538d2e3b57ada6dd8a24fb147c10ef24ae8c52b18470e4" gracePeriod=30 Oct 30 13:18:41.002370 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:41.002328 4931 patch_prober.go:29] interesting pod/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:18:41.002591 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:41.002410 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" podUID="9ef8a128-1b22-4ef2-81ed-0a1b494393c3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:18:44.802102 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:44.799607 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:18:45.553969 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.553924 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:18:45.561512 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.561478 4931 generic.go:356] "Generic (PLEG): container finished" podID="9ef8a128-1b22-4ef2-81ed-0a1b494393c3" containerID="528bea8ad0c34002c1538d2e3b57ada6dd8a24fb147c10ef24ae8c52b18470e4" exitCode=0 Oct 30 13:18:45.561665 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.561529 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" Oct 30 13:18:45.561665 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.561542 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" event={"ID":"9ef8a128-1b22-4ef2-81ed-0a1b494393c3","Type":"ContainerDied","Data":"528bea8ad0c34002c1538d2e3b57ada6dd8a24fb147c10ef24ae8c52b18470e4"} Oct 30 13:18:45.561665 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.561603 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf" event={"ID":"9ef8a128-1b22-4ef2-81ed-0a1b494393c3","Type":"ContainerDied","Data":"2ce037ab67784ab8b32f52041e5a4ba783eb2dd80af610364221325ac529302f"} Oct 30 13:18:45.561665 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.561628 4931 scope.go:122] "RemoveContainer" containerID="528bea8ad0c34002c1538d2e3b57ada6dd8a24fb147c10ef24ae8c52b18470e4" Oct 30 13:18:45.572801 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.572781 4931 scope.go:122] "RemoveContainer" containerID="4e1eeee4d82f56f22f06c4a0c2ed45a49a1fb852be0f2d410ed0f0ed98784c6e" Oct 30 13:18:45.584374 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.584355 4931 scope.go:122] "RemoveContainer" containerID="528bea8ad0c34002c1538d2e3b57ada6dd8a24fb147c10ef24ae8c52b18470e4" Oct 30 13:18:45.584699 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:18:45.584678 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"528bea8ad0c34002c1538d2e3b57ada6dd8a24fb147c10ef24ae8c52b18470e4\": container with ID starting with 528bea8ad0c34002c1538d2e3b57ada6dd8a24fb147c10ef24ae8c52b18470e4 not found: ID does not exist" containerID="528bea8ad0c34002c1538d2e3b57ada6dd8a24fb147c10ef24ae8c52b18470e4" Oct 30 13:18:45.584801 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.584706 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"528bea8ad0c34002c1538d2e3b57ada6dd8a24fb147c10ef24ae8c52b18470e4"} err="failed to get container status \"528bea8ad0c34002c1538d2e3b57ada6dd8a24fb147c10ef24ae8c52b18470e4\": rpc error: code = NotFound desc = could not find container \"528bea8ad0c34002c1538d2e3b57ada6dd8a24fb147c10ef24ae8c52b18470e4\": container with ID starting with 528bea8ad0c34002c1538d2e3b57ada6dd8a24fb147c10ef24ae8c52b18470e4 not found: ID does not exist" Oct 30 13:18:45.584801 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.584723 4931 scope.go:122] "RemoveContainer" containerID="4e1eeee4d82f56f22f06c4a0c2ed45a49a1fb852be0f2d410ed0f0ed98784c6e" Oct 30 13:18:45.584993 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:18:45.584973 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e1eeee4d82f56f22f06c4a0c2ed45a49a1fb852be0f2d410ed0f0ed98784c6e\": container with ID starting with 4e1eeee4d82f56f22f06c4a0c2ed45a49a1fb852be0f2d410ed0f0ed98784c6e not found: ID does not exist" containerID="4e1eeee4d82f56f22f06c4a0c2ed45a49a1fb852be0f2d410ed0f0ed98784c6e" Oct 30 13:18:45.585048 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.585004 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e1eeee4d82f56f22f06c4a0c2ed45a49a1fb852be0f2d410ed0f0ed98784c6e"} err="failed to get container status \"4e1eeee4d82f56f22f06c4a0c2ed45a49a1fb852be0f2d410ed0f0ed98784c6e\": rpc error: code = NotFound desc = could not find container \"4e1eeee4d82f56f22f06c4a0c2ed45a49a1fb852be0f2d410ed0f0ed98784c6e\": container with ID starting with 4e1eeee4d82f56f22f06c4a0c2ed45a49a1fb852be0f2d410ed0f0ed98784c6e not found: ID does not exist" Oct 30 13:18:45.660742 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.658254 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-istio-podinfo\") pod \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " Oct 30 13:18:45.660742 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.658485 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-workload-socket\") pod \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " Oct 30 13:18:45.660742 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.658616 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-istio-token\" (UniqueName: \"kubernetes.io/projected/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-istio-token\") pod \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " Oct 30 13:18:45.660742 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.658683 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-kube-api-access-sm7sw\" (UniqueName: \"kubernetes.io/projected/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-kube-api-access-sm7sw\") pod \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " Oct 30 13:18:45.660742 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.658714 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-istio-data\") pod \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " Oct 30 13:18:45.660742 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.658811 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-workload-certs\") pod \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " Oct 30 13:18:45.660742 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.658844 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-credential-socket\") pod \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " Oct 30 13:18:45.660742 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.658874 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-istiod-ca-cert\") pod \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " Oct 30 13:18:45.660742 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.659021 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-istio-data" pod "9ef8a128-1b22-4ef2-81ed-0a1b494393c3" (UID: "9ef8a128-1b22-4ef2-81ed-0a1b494393c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:45.660742 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.659035 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-istio-envoy\") pod \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\" (UID: \"9ef8a128-1b22-4ef2-81ed-0a1b494393c3\") " Oct 30 13:18:45.660742 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.659196 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-workload-certs" pod "9ef8a128-1b22-4ef2-81ed-0a1b494393c3" (UID: "9ef8a128-1b22-4ef2-81ed-0a1b494393c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:45.660742 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.659407 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-workload-socket" pod "9ef8a128-1b22-4ef2-81ed-0a1b494393c3" (UID: "9ef8a128-1b22-4ef2-81ed-0a1b494393c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:45.660742 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.659434 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-credential-socket" pod "9ef8a128-1b22-4ef2-81ed-0a1b494393c3" (UID: "9ef8a128-1b22-4ef2-81ed-0a1b494393c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:45.660742 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.659891 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:18:45.660742 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.659912 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:18:45.660742 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.659927 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:18:45.660742 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.659942 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:18:45.660742 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.660501 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-istiod-ca-cert" pod "9ef8a128-1b22-4ef2-81ed-0a1b494393c3" (UID: "9ef8a128-1b22-4ef2-81ed-0a1b494393c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:18:45.663224 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.663173 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-kube-api-access-sm7sw" pod "9ef8a128-1b22-4ef2-81ed-0a1b494393c3" (UID: "9ef8a128-1b22-4ef2-81ed-0a1b494393c3"). InnerVolumeSpecName "kube-api-access-sm7sw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:45.663224 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.663186 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-istio-envoy" pod "9ef8a128-1b22-4ef2-81ed-0a1b494393c3" (UID: "9ef8a128-1b22-4ef2-81ed-0a1b494393c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:45.663463 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.663256 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-istio-podinfo" pod "9ef8a128-1b22-4ef2-81ed-0a1b494393c3" (UID: "9ef8a128-1b22-4ef2-81ed-0a1b494393c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:18:45.669500 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.669471 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-istio-token" pod "9ef8a128-1b22-4ef2-81ed-0a1b494393c3" (UID: "9ef8a128-1b22-4ef2-81ed-0a1b494393c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:45.761456 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.761374 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:18:45.761456 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.761410 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:18:45.761456 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.761424 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:18:45.761456 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.761439 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm7sw\" (UniqueName: \"kubernetes.io/projected/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-kube-api-access-sm7sw\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:18:45.761456 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.761454 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ef8a128-1b22-4ef2-81ed-0a1b494393c3-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:18:45.916843 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.916806 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf"] Oct 30 13:18:45.930123 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:45.930089 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf"] Oct 30 13:18:47.516073 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:18:47.516042 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ef8a128-1b22-4ef2-81ed-0a1b494393c3" path="/var/lib/kubelet/pods/9ef8a128-1b22-4ef2-81ed-0a1b494393c3/volumes" Oct 30 13:19:00.955968 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:00.955925 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8"] Oct 30 13:19:00.956949 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:00.956923 4931 cpu_manager.go:414] "containerMap: removing container" podUID="9ef8a128-1b22-4ef2-81ed-0a1b494393c3" containerName="istio-proxy" Oct 30 13:19:00.957045 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:00.956953 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ef8a128-1b22-4ef2-81ed-0a1b494393c3" containerName="istio-proxy" Oct 30 13:19:00.957045 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:00.956972 4931 cpu_manager.go:414] "containerMap: removing container" podUID="9ef8a128-1b22-4ef2-81ed-0a1b494393c3" containerName="istio-proxy" Oct 30 13:19:00.957045 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:00.956983 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ef8a128-1b22-4ef2-81ed-0a1b494393c3" containerName="istio-proxy" Oct 30 13:19:00.957205 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:00.957176 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ef8a128-1b22-4ef2-81ed-0a1b494393c3" containerName="istio-proxy" Oct 30 13:19:00.957205 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:00.957189 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ef8a128-1b22-4ef2-81ed-0a1b494393c3" containerName="istio-proxy" Oct 30 13:19:00.961613 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:00.961587 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:19:00.986526 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:00.986490 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8"] Oct 30 13:19:01.037653 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.037580 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdk6s\" (UniqueName: \"kubernetes.io/projected/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-kube-api-access-fdk6s\") pod \"e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:19:01.037653 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.037644 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-istio-data\") pod \"e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:19:01.037927 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.037787 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-istiod-ca-cert\") pod \"e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:19:01.037927 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.037863 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-istio-podinfo\") pod \"e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:19:01.037927 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.037905 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-istio-envoy\") pod \"e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:19:01.038611 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.037931 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-credential-socket\") pod \"e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:19:01.038611 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.038001 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-istio-token\") pod \"e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:19:01.038611 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.038051 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-workload-certs\") pod \"e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:19:01.038611 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.038186 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-workload-socket\") pod \"e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:19:01.040104 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.040061 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-619b9b-openshift-default-dockercfg-kp5t6\"" Oct 30 13:19:01.139673 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.139639 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdk6s\" (UniqueName: \"kubernetes.io/projected/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-kube-api-access-fdk6s\") pod \"e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:19:01.139673 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.139685 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-istio-data\") pod \"e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:19:01.139923 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.139819 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-istiod-ca-cert\") pod \"e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:19:01.139923 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.139903 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-istio-podinfo\") pod \"e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:19:01.140022 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.139933 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-istio-envoy\") pod \"e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:19:01.140022 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.139961 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-credential-socket\") pod \"e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:19:01.140107 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.140072 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-istio-token\") pod \"e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:19:01.140159 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.140128 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-workload-certs\") pod \"e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:19:01.140261 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.140246 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-workload-socket\") pod \"e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:19:01.140443 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.140421 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-istio-data\") pod \"e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:19:01.140543 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.140523 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-credential-socket\") pod \"e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:19:01.140618 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.140592 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-workload-certs\") pod \"e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:19:01.140681 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.140626 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-workload-socket\") pod \"e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:19:01.140998 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.140975 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-istiod-ca-cert\") pod \"e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:19:01.142403 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.142379 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-istio-envoy\") pod \"e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:19:01.142635 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.142619 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-istio-podinfo\") pod \"e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:19:01.166666 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.166631 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-istio-token\") pod \"e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:19:01.167479 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.167461 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdk6s\" (UniqueName: \"kubernetes.io/projected/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-kube-api-access-fdk6s\") pod \"e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:19:01.282391 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.279099 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:19:01.702870 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.702843 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8"] Oct 30 13:19:01.705026 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:19:01.704991 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod212428c9_b6b3_4757_9ef5_ffe8b2fa0dbf.slice/crio-8ba002a8ac598fb1491ed425d29cf32172f4a7721a5386ae6d067de49da4edf1 WatchSource:0}: Error finding container 8ba002a8ac598fb1491ed425d29cf32172f4a7721a5386ae6d067de49da4edf1: Status 404 returned error can't find the container with id 8ba002a8ac598fb1491ed425d29cf32172f4a7721a5386ae6d067de49da4edf1 Oct 30 13:19:01.707967 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.707930 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:19:01.708073 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.708000 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:19:01.708073 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.708028 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:19:01.873726 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.873695 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" event={"ID":"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf","Type":"ContainerStarted","Data":"f37ca073e38d4e6229125b4e00a5e9b5222822eaa461c4de7621af75389f1f98"} Oct 30 13:19:01.873936 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.873749 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" event={"ID":"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf","Type":"ContainerStarted","Data":"8ba002a8ac598fb1491ed425d29cf32172f4a7721a5386ae6d067de49da4edf1"} Oct 30 13:19:01.924146 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:01.924081 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" podStartSLOduration=1.9240637619999998 podStartE2EDuration="1.924063762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:01.921789317 +0000 UTC m=+2040.525287774" watchObservedRunningTime="2026-10-30 13:19:01.924063762 +0000 UTC m=+2040.527562280" Oct 30 13:19:02.280190 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:02.280153 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:19:02.286691 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:02.286661 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:19:02.681326 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:02.681231 4931 scope.go:122] "RemoveContainer" containerID="25a1941ee4642a72828a9cd2b8c95a011309390c8b7719a05c8a451fb485eb3f" Oct 30 13:19:02.693023 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:02.692999 4931 scope.go:122] "RemoveContainer" containerID="b127bd0664b664dd8e588f6157e40f2e32cc940aaa6bad77c5bc22d781be44a5" Oct 30 13:19:02.882775 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:02.882740 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:19:02.883820 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:02.883795 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:19:07.354762 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:19:07.354719 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:19:12.156368 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:12.156328 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77dd3317-nwgz6"] Oct 30 13:19:12.162562 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:12.162535 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dd3317-nwgz6" Oct 30 13:19:12.190663 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:12.190589 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77dd3317-nwgz6"] Oct 30 13:19:12.201073 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:12.201034 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:19:12.201508 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:12.201481 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:19:12.225369 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:12.225331 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dce7a7cd-c031-47f5-af51-5fdbe4f9e435-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77dd3317-nwgz6\" (UID: \"dce7a7cd-c031-47f5-af51-5fdbe4f9e435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dd3317-nwgz6" Oct 30 13:19:12.225665 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:12.225634 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dce7a7cd-c031-47f5-af51-5fdbe4f9e435-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77dd3317-nwgz6\" (UID: \"dce7a7cd-c031-47f5-af51-5fdbe4f9e435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dd3317-nwgz6" Oct 30 13:19:12.226521 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:12.225681 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zv7q\" (UniqueName: \"kubernetes.io/projected/dce7a7cd-c031-47f5-af51-5fdbe4f9e435-kube-api-access-7zv7q\") pod \"maas-api-revoke-sub-77dd3317-nwgz6\" (UID: \"dce7a7cd-c031-47f5-af51-5fdbe4f9e435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dd3317-nwgz6" Oct 30 13:19:12.327489 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:12.327439 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dce7a7cd-c031-47f5-af51-5fdbe4f9e435-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77dd3317-nwgz6\" (UID: \"dce7a7cd-c031-47f5-af51-5fdbe4f9e435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dd3317-nwgz6" Oct 30 13:19:12.327717 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:12.327554 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dce7a7cd-c031-47f5-af51-5fdbe4f9e435-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77dd3317-nwgz6\" (UID: \"dce7a7cd-c031-47f5-af51-5fdbe4f9e435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dd3317-nwgz6" Oct 30 13:19:12.327717 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:12.327589 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zv7q\" (UniqueName: \"kubernetes.io/projected/dce7a7cd-c031-47f5-af51-5fdbe4f9e435-kube-api-access-7zv7q\") pod \"maas-api-revoke-sub-77dd3317-nwgz6\" (UID: \"dce7a7cd-c031-47f5-af51-5fdbe4f9e435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dd3317-nwgz6" Oct 30 13:19:12.328210 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:12.328180 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dce7a7cd-c031-47f5-af51-5fdbe4f9e435-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77dd3317-nwgz6\" (UID: \"dce7a7cd-c031-47f5-af51-5fdbe4f9e435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dd3317-nwgz6" Oct 30 13:19:12.330049 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:12.330024 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dce7a7cd-c031-47f5-af51-5fdbe4f9e435-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77dd3317-nwgz6\" (UID: \"dce7a7cd-c031-47f5-af51-5fdbe4f9e435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dd3317-nwgz6" Oct 30 13:19:12.354835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:12.354809 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zv7q\" (UniqueName: \"kubernetes.io/projected/dce7a7cd-c031-47f5-af51-5fdbe4f9e435-kube-api-access-7zv7q\") pod \"maas-api-revoke-sub-77dd3317-nwgz6\" (UID: \"dce7a7cd-c031-47f5-af51-5fdbe4f9e435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dd3317-nwgz6" Oct 30 13:19:12.478472 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:12.478416 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dd3317-nwgz6" Oct 30 13:19:12.794870 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:12.794844 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77dd3317-nwgz6"] Oct 30 13:19:12.795611 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:19:12.795582 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce7a7cd_c031_47f5_af51_5fdbe4f9e435.slice/crio-094544639da8e8aa9008eb8fb99411b4cac284fae18993a84fc306d1df1fec10 WatchSource:0}: Error finding container 094544639da8e8aa9008eb8fb99411b4cac284fae18993a84fc306d1df1fec10: Status 404 returned error can't find the container with id 094544639da8e8aa9008eb8fb99411b4cac284fae18993a84fc306d1df1fec10 Oct 30 13:19:12.954413 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:12.954378 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2"] Oct 30 13:19:12.954630 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:12.954604 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" podUID="d3a9c9f2-eaac-4fcc-88e7-87d311d77677" containerName="istio-proxy" containerID="cri-o://1ea9cf76063beed45e1a77f55f59edba70264cb55533b7cf544982535626f98c" gracePeriod=30 Oct 30 13:19:12.956659 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:12.956629 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dd3317-nwgz6" event={"ID":"dce7a7cd-c031-47f5-af51-5fdbe4f9e435","Type":"ContainerStarted","Data":"094544639da8e8aa9008eb8fb99411b4cac284fae18993a84fc306d1df1fec10"} Oct 30 13:19:13.879859 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:13.879824 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj"] Oct 30 13:19:13.880243 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:13.880016 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" podUID="e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6" containerName="istio-proxy" containerID="cri-o://d017bb2161ab1ad8e3f37effa68a5f3c5f626dfaeecfba6342eddd0ebda6585e" gracePeriod=30 Oct 30 13:19:13.969490 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:13.969451 4931 generic.go:356] "Generic (PLEG): container finished" podID="dce7a7cd-c031-47f5-af51-5fdbe4f9e435" containerID="9226e5e4f51d6432d95b7ae9cfb12ec5cde6f93143829645f99958feab6b8276" exitCode=0 Oct 30 13:19:13.969809 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:13.969785 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dd3317-nwgz6" event={"ID":"dce7a7cd-c031-47f5-af51-5fdbe4f9e435","Type":"ContainerDied","Data":"9226e5e4f51d6432d95b7ae9cfb12ec5cde6f93143829645f99958feab6b8276"} Oct 30 13:19:15.158768 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:15.158738 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dd3317-nwgz6" Oct 30 13:19:15.269354 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:15.269233 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dce7a7cd-c031-47f5-af51-5fdbe4f9e435-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dce7a7cd-c031-47f5-af51-5fdbe4f9e435-maas-api-xks-ca\") pod \"dce7a7cd-c031-47f5-af51-5fdbe4f9e435\" (UID: \"dce7a7cd-c031-47f5-af51-5fdbe4f9e435\") " Oct 30 13:19:15.269520 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:15.269430 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dce7a7cd-c031-47f5-af51-5fdbe4f9e435-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dce7a7cd-c031-47f5-af51-5fdbe4f9e435-maas-api-openshift-ca\") pod \"dce7a7cd-c031-47f5-af51-5fdbe4f9e435\" (UID: \"dce7a7cd-c031-47f5-af51-5fdbe4f9e435\") " Oct 30 13:19:15.269520 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:15.269508 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dce7a7cd-c031-47f5-af51-5fdbe4f9e435-kube-api-access-7zv7q\" (UniqueName: \"kubernetes.io/projected/dce7a7cd-c031-47f5-af51-5fdbe4f9e435-kube-api-access-7zv7q\") pod \"dce7a7cd-c031-47f5-af51-5fdbe4f9e435\" (UID: \"dce7a7cd-c031-47f5-af51-5fdbe4f9e435\") " Oct 30 13:19:15.269874 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:15.269822 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce7a7cd-c031-47f5-af51-5fdbe4f9e435-maas-api-openshift-ca" pod "dce7a7cd-c031-47f5-af51-5fdbe4f9e435" (UID: "dce7a7cd-c031-47f5-af51-5fdbe4f9e435"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:19:15.270151 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:15.270131 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dce7a7cd-c031-47f5-af51-5fdbe4f9e435-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:19:15.271561 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:15.271527 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dce7a7cd-c031-47f5-af51-5fdbe4f9e435-maas-api-xks-ca" pod "dce7a7cd-c031-47f5-af51-5fdbe4f9e435" (UID: "dce7a7cd-c031-47f5-af51-5fdbe4f9e435"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:19:15.271750 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:15.271726 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce7a7cd-c031-47f5-af51-5fdbe4f9e435-kube-api-access-7zv7q" pod "dce7a7cd-c031-47f5-af51-5fdbe4f9e435" (UID: "dce7a7cd-c031-47f5-af51-5fdbe4f9e435"). InnerVolumeSpecName "kube-api-access-7zv7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:19:15.371983 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:15.371948 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dce7a7cd-c031-47f5-af51-5fdbe4f9e435-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:19:15.371983 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:15.371982 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zv7q\" (UniqueName: \"kubernetes.io/projected/dce7a7cd-c031-47f5-af51-5fdbe4f9e435-kube-api-access-7zv7q\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:19:15.991932 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:15.991888 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dd3317-nwgz6" Oct 30 13:19:15.992152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:15.991898 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dd3317-nwgz6" event={"ID":"dce7a7cd-c031-47f5-af51-5fdbe4f9e435","Type":"ContainerDied","Data":"094544639da8e8aa9008eb8fb99411b4cac284fae18993a84fc306d1df1fec10"} Oct 30 13:19:15.992152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:15.992036 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="094544639da8e8aa9008eb8fb99411b4cac284fae18993a84fc306d1df1fec10" Oct 30 13:19:17.644402 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:17.644366 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9"] Oct 30 13:19:17.645218 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:17.645194 4931 cpu_manager.go:414] "containerMap: removing container" podUID="dce7a7cd-c031-47f5-af51-5fdbe4f9e435" containerName="revoke-keys" Oct 30 13:19:17.645363 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:17.645219 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dce7a7cd-c031-47f5-af51-5fdbe4f9e435" containerName="revoke-keys" Oct 30 13:19:17.645426 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:17.645396 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="dce7a7cd-c031-47f5-af51-5fdbe4f9e435" containerName="revoke-keys" Oct 30 13:19:17.649200 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:17.649180 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9" Oct 30 13:19:17.678134 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:17.678106 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-60a9a6-serving-cert\"" Oct 30 13:19:17.685322 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:17.685288 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9"] Oct 30 13:19:17.801022 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:17.800982 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd"] Oct 30 13:19:17.806457 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:17.806423 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd" Oct 30 13:19:17.811279 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:17.811235 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jgbd\" (UniqueName: \"kubernetes.io/projected/1a6eeedf-7968-49d9-a1f0-d8291c719f8c-kube-api-access-7jgbd\") pod \"maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9\" (UID: \"1a6eeedf-7968-49d9-a1f0-d8291c719f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9" Oct 30 13:19:17.811421 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:17.811304 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a6eeedf-7968-49d9-a1f0-d8291c719f8c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9\" (UID: \"1a6eeedf-7968-49d9-a1f0-d8291c719f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9" Oct 30 13:19:17.811470 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:17.811425 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a6eeedf-7968-49d9-a1f0-d8291c719f8c-metrics-certs\") pod \"maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9\" (UID: \"1a6eeedf-7968-49d9-a1f0-d8291c719f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9" Oct 30 13:19:17.811678 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:17.811656 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a6eeedf-7968-49d9-a1f0-d8291c719f8c-maas-api-tls\") pod \"maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9\" (UID: \"1a6eeedf-7968-49d9-a1f0-d8291c719f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9" Oct 30 13:19:17.826583 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:17.826552 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd"] Oct 30 13:19:17.834631 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:17.834601 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cd2d2e\"/\"test-model-cd2d2e-kserve-self-signed-certs\"" Oct 30 13:19:17.834744 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:17.834722 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cd2d2e\"/\"kube-root-ca.crt\"" Oct 30 13:19:17.834824 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:17.834809 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cd2d2e\"/\"openshift-service-ca.crt\"" Oct 30 13:19:17.834925 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:17.834910 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cd2d2e\"/\"default-dockercfg-pv4rf\"" Oct 30 13:19:17.910771 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:17.910684 4931 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:19:17.910771 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:17.910753 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" podUID="d3a9c9f2-eaac-4fcc-88e7-87d311d77677" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:19:17.913713 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:17.913680 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a6eeedf-7968-49d9-a1f0-d8291c719f8c-maas-api-tls\") pod \"maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9\" (UID: \"1a6eeedf-7968-49d9-a1f0-d8291c719f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9" Oct 30 13:19:17.913838 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:17.913787 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2023086a-0026-41a0-9180-d48b4523e392-home\") pod \"test-model-cd2d2e-kserve-7d8c46bfbc-2htxd\" (UID: \"2023086a-0026-41a0-9180-d48b4523e392\") " pod="ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd" Oct 30 13:19:17.913838 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:17.913833 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2023086a-0026-41a0-9180-d48b4523e392-dshm\") pod \"test-model-cd2d2e-kserve-7d8c46bfbc-2htxd\" (UID: \"2023086a-0026-41a0-9180-d48b4523e392\") " pod="ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd" Oct 30 13:19:17.913992 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:17.913975 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jgbd\" (UniqueName: \"kubernetes.io/projected/1a6eeedf-7968-49d9-a1f0-d8291c719f8c-kube-api-access-7jgbd\") pod \"maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9\" (UID: \"1a6eeedf-7968-49d9-a1f0-d8291c719f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9" Oct 30 13:19:17.914053 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:17.914012 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a6eeedf-7968-49d9-a1f0-d8291c719f8c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9\" (UID: \"1a6eeedf-7968-49d9-a1f0-d8291c719f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9" Oct 30 13:19:17.914141 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:17.914113 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a6eeedf-7968-49d9-a1f0-d8291c719f8c-metrics-certs\") pod \"maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9\" (UID: \"1a6eeedf-7968-49d9-a1f0-d8291c719f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9" Oct 30 13:19:17.914206 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:17.914192 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2023086a-0026-41a0-9180-d48b4523e392-tmp-dir\") pod \"test-model-cd2d2e-kserve-7d8c46bfbc-2htxd\" (UID: \"2023086a-0026-41a0-9180-d48b4523e392\") " pod="ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd" Oct 30 13:19:17.914303 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:17.914288 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2023086a-0026-41a0-9180-d48b4523e392-model-cache\") pod \"test-model-cd2d2e-kserve-7d8c46bfbc-2htxd\" (UID: \"2023086a-0026-41a0-9180-d48b4523e392\") " pod="ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd" Oct 30 13:19:17.914370 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:17.914342 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xt4v\" (UniqueName: \"kubernetes.io/projected/2023086a-0026-41a0-9180-d48b4523e392-kube-api-access-2xt4v\") pod \"test-model-cd2d2e-kserve-7d8c46bfbc-2htxd\" (UID: \"2023086a-0026-41a0-9180-d48b4523e392\") " pod="ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd" Oct 30 13:19:17.914505 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:17.914484 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2023086a-0026-41a0-9180-d48b4523e392-tls-certs\") pod \"test-model-cd2d2e-kserve-7d8c46bfbc-2htxd\" (UID: \"2023086a-0026-41a0-9180-d48b4523e392\") " pod="ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd" Oct 30 13:19:17.914859 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:17.914828 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a6eeedf-7968-49d9-a1f0-d8291c719f8c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9\" (UID: \"1a6eeedf-7968-49d9-a1f0-d8291c719f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9" Oct 30 13:19:17.916825 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:17.916796 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a6eeedf-7968-49d9-a1f0-d8291c719f8c-metrics-certs\") pod \"maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9\" (UID: \"1a6eeedf-7968-49d9-a1f0-d8291c719f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9" Oct 30 13:19:17.916981 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:17.916961 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a6eeedf-7968-49d9-a1f0-d8291c719f8c-maas-api-tls\") pod \"maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9\" (UID: \"1a6eeedf-7968-49d9-a1f0-d8291c719f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9" Oct 30 13:19:17.939157 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:17.939129 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jgbd\" (UniqueName: \"kubernetes.io/projected/1a6eeedf-7968-49d9-a1f0-d8291c719f8c-kube-api-access-7jgbd\") pod \"maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9\" (UID: \"1a6eeedf-7968-49d9-a1f0-d8291c719f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9" Oct 30 13:19:17.964036 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:17.963984 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9" Oct 30 13:19:18.016443 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.016406 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2023086a-0026-41a0-9180-d48b4523e392-home\") pod \"test-model-cd2d2e-kserve-7d8c46bfbc-2htxd\" (UID: \"2023086a-0026-41a0-9180-d48b4523e392\") " pod="ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd" Oct 30 13:19:18.016578 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.016472 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2023086a-0026-41a0-9180-d48b4523e392-dshm\") pod \"test-model-cd2d2e-kserve-7d8c46bfbc-2htxd\" (UID: \"2023086a-0026-41a0-9180-d48b4523e392\") " pod="ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd" Oct 30 13:19:18.016655 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.016604 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2023086a-0026-41a0-9180-d48b4523e392-tmp-dir\") pod \"test-model-cd2d2e-kserve-7d8c46bfbc-2htxd\" (UID: \"2023086a-0026-41a0-9180-d48b4523e392\") " pod="ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd" Oct 30 13:19:18.016717 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.016667 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2023086a-0026-41a0-9180-d48b4523e392-model-cache\") pod \"test-model-cd2d2e-kserve-7d8c46bfbc-2htxd\" (UID: \"2023086a-0026-41a0-9180-d48b4523e392\") " pod="ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd" Oct 30 13:19:18.016717 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.016709 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xt4v\" (UniqueName: \"kubernetes.io/projected/2023086a-0026-41a0-9180-d48b4523e392-kube-api-access-2xt4v\") pod \"test-model-cd2d2e-kserve-7d8c46bfbc-2htxd\" (UID: \"2023086a-0026-41a0-9180-d48b4523e392\") " pod="ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd" Oct 30 13:19:18.016824 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.016780 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2023086a-0026-41a0-9180-d48b4523e392-tls-certs\") pod \"test-model-cd2d2e-kserve-7d8c46bfbc-2htxd\" (UID: \"2023086a-0026-41a0-9180-d48b4523e392\") " pod="ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd" Oct 30 13:19:18.016882 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.016854 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2023086a-0026-41a0-9180-d48b4523e392-home\") pod \"test-model-cd2d2e-kserve-7d8c46bfbc-2htxd\" (UID: \"2023086a-0026-41a0-9180-d48b4523e392\") " pod="ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd" Oct 30 13:19:18.017349 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.017205 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2023086a-0026-41a0-9180-d48b4523e392-model-cache\") pod \"test-model-cd2d2e-kserve-7d8c46bfbc-2htxd\" (UID: \"2023086a-0026-41a0-9180-d48b4523e392\") " pod="ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd" Oct 30 13:19:18.017349 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.017306 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2023086a-0026-41a0-9180-d48b4523e392-tmp-dir\") pod \"test-model-cd2d2e-kserve-7d8c46bfbc-2htxd\" (UID: \"2023086a-0026-41a0-9180-d48b4523e392\") " pod="ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd" Oct 30 13:19:18.019148 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.019125 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2023086a-0026-41a0-9180-d48b4523e392-dshm\") pod \"test-model-cd2d2e-kserve-7d8c46bfbc-2htxd\" (UID: \"2023086a-0026-41a0-9180-d48b4523e392\") " pod="ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd" Oct 30 13:19:18.019595 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.019576 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2023086a-0026-41a0-9180-d48b4523e392-tls-certs\") pod \"test-model-cd2d2e-kserve-7d8c46bfbc-2htxd\" (UID: \"2023086a-0026-41a0-9180-d48b4523e392\") " pod="ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd" Oct 30 13:19:18.055586 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.045511 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xt4v\" (UniqueName: \"kubernetes.io/projected/2023086a-0026-41a0-9180-d48b4523e392-kube-api-access-2xt4v\") pod \"test-model-cd2d2e-kserve-7d8c46bfbc-2htxd\" (UID: \"2023086a-0026-41a0-9180-d48b4523e392\") " pod="ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd" Oct 30 13:19:18.153189 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.153151 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd" Oct 30 13:19:18.321833 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.321339 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9"] Oct 30 13:19:18.456781 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.456756 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" Oct 30 13:19:18.534371 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.534340 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd"] Oct 30 13:19:18.535814 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:19:18.535784 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2023086a_0026_41a0_9180_d48b4523e392.slice/crio-690c0a4b0ad153bc0b1001316da330d993879377eb9d89ccdbdc0075a68071dd WatchSource:0}: Error finding container 690c0a4b0ad153bc0b1001316da330d993879377eb9d89ccdbdc0075a68071dd: Status 404 returned error can't find the container with id 690c0a4b0ad153bc0b1001316da330d993879377eb9d89ccdbdc0075a68071dd Oct 30 13:19:18.575613 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.574245 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-istio-envoy\") pod \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " Oct 30 13:19:18.575613 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.574347 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-workload-socket\") pod \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " Oct 30 13:19:18.575613 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.574455 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-istio-token\" (UniqueName: \"kubernetes.io/projected/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-istio-token\") pod \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " Oct 30 13:19:18.575613 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.574501 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-istiod-ca-cert\") pod \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " Oct 30 13:19:18.575613 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.574563 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-kube-api-access-s7h2s\" (UniqueName: \"kubernetes.io/projected/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-kube-api-access-s7h2s\") pod \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " Oct 30 13:19:18.575613 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.574616 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-workload-certs\") pod \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " Oct 30 13:19:18.575613 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.574715 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-istio-podinfo\") pod \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " Oct 30 13:19:18.575613 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.574748 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-istio-data\") pod \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " Oct 30 13:19:18.575613 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.574925 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-credential-socket\") pod \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\" (UID: \"d3a9c9f2-eaac-4fcc-88e7-87d311d77677\") " Oct 30 13:19:18.575613 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.575369 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-istiod-ca-cert" pod "d3a9c9f2-eaac-4fcc-88e7-87d311d77677" (UID: "d3a9c9f2-eaac-4fcc-88e7-87d311d77677"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:19:18.576210 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.575743 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:19:18.576399 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.576364 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-credential-socket" pod "d3a9c9f2-eaac-4fcc-88e7-87d311d77677" (UID: "d3a9c9f2-eaac-4fcc-88e7-87d311d77677"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:18.577187 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.577159 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-istio-data" pod "d3a9c9f2-eaac-4fcc-88e7-87d311d77677" (UID: "d3a9c9f2-eaac-4fcc-88e7-87d311d77677"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:18.577417 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.577392 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-workload-certs" pod "d3a9c9f2-eaac-4fcc-88e7-87d311d77677" (UID: "d3a9c9f2-eaac-4fcc-88e7-87d311d77677"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:18.577950 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.577925 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-workload-socket" pod "d3a9c9f2-eaac-4fcc-88e7-87d311d77677" (UID: "d3a9c9f2-eaac-4fcc-88e7-87d311d77677"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:18.578693 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.578672 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-istio-envoy" pod "d3a9c9f2-eaac-4fcc-88e7-87d311d77677" (UID: "d3a9c9f2-eaac-4fcc-88e7-87d311d77677"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:18.582506 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.582479 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-istio-token" pod "d3a9c9f2-eaac-4fcc-88e7-87d311d77677" (UID: "d3a9c9f2-eaac-4fcc-88e7-87d311d77677"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:19:18.582613 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.582520 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-kube-api-access-s7h2s" pod "d3a9c9f2-eaac-4fcc-88e7-87d311d77677" (UID: "d3a9c9f2-eaac-4fcc-88e7-87d311d77677"). InnerVolumeSpecName "kube-api-access-s7h2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:19:18.590438 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.590406 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-istio-podinfo" pod "d3a9c9f2-eaac-4fcc-88e7-87d311d77677" (UID: "d3a9c9f2-eaac-4fcc-88e7-87d311d77677"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:19:18.695251 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.693687 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:19:18.695251 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.694627 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:19:18.695251 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.694650 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:19:18.695251 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.694664 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:19:18.695251 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.694678 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7h2s\" (UniqueName: \"kubernetes.io/projected/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-kube-api-access-s7h2s\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:19:18.695251 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.694693 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:19:18.695251 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.694707 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:19:18.695251 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:18.694718 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3a9c9f2-eaac-4fcc-88e7-87d311d77677-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:19:19.008174 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.008140 4931 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 13:19:19.008361 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.008200 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" podUID="e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 13:19:19.034811 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.027106 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9" event={"ID":"1a6eeedf-7968-49d9-a1f0-d8291c719f8c","Type":"ContainerStarted","Data":"684ebce8f863e5de3c8d24e18b8758096225ce0dc50fafe674e89fff885638f7"} Oct 30 13:19:19.034811 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.027170 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9" event={"ID":"1a6eeedf-7968-49d9-a1f0-d8291c719f8c","Type":"ContainerStarted","Data":"ce78611cbc405f5cf2b3ec69aaebc0df31c9097e9c5f8a1ef83a3fba2edc08bf"} Oct 30 13:19:19.034811 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.028257 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9" Oct 30 13:19:19.034811 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.033832 4931 generic.go:356] "Generic (PLEG): container finished" podID="d3a9c9f2-eaac-4fcc-88e7-87d311d77677" containerID="1ea9cf76063beed45e1a77f55f59edba70264cb55533b7cf544982535626f98c" exitCode=0 Oct 30 13:19:19.034811 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.033957 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" event={"ID":"d3a9c9f2-eaac-4fcc-88e7-87d311d77677","Type":"ContainerDied","Data":"1ea9cf76063beed45e1a77f55f59edba70264cb55533b7cf544982535626f98c"} Oct 30 13:19:19.034811 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.034006 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" event={"ID":"d3a9c9f2-eaac-4fcc-88e7-87d311d77677","Type":"ContainerDied","Data":"60d0d8b9354cd986d31027673571709fea97eec15479bde577b022015e9f8b6c"} Oct 30 13:19:19.034811 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.034035 4931 scope.go:122] "RemoveContainer" containerID="1ea9cf76063beed45e1a77f55f59edba70264cb55533b7cf544982535626f98c" Oct 30 13:19:19.034811 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.034196 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2" Oct 30 13:19:19.054340 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.048612 4931 generic.go:356] "Generic (PLEG): container finished" podID="e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6" containerID="d017bb2161ab1ad8e3f37effa68a5f3c5f626dfaeecfba6342eddd0ebda6585e" exitCode=0 Oct 30 13:19:19.054340 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.048696 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" event={"ID":"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6","Type":"ContainerDied","Data":"d017bb2161ab1ad8e3f37effa68a5f3c5f626dfaeecfba6342eddd0ebda6585e"} Oct 30 13:19:19.054340 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.051030 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd" event={"ID":"2023086a-0026-41a0-9180-d48b4523e392","Type":"ContainerStarted","Data":"ce620dd132925e305485ef5ffdcd064e3e31d7d27def84562213dfff4ec4ea12"} Oct 30 13:19:19.054340 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.051067 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd" event={"ID":"2023086a-0026-41a0-9180-d48b4523e392","Type":"ContainerStarted","Data":"690c0a4b0ad153bc0b1001316da330d993879377eb9d89ccdbdc0075a68071dd"} Oct 30 13:19:19.063350 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.063303 4931 scope.go:122] "RemoveContainer" containerID="1ea9cf76063beed45e1a77f55f59edba70264cb55533b7cf544982535626f98c" Oct 30 13:19:19.064336 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:19:19.063702 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ea9cf76063beed45e1a77f55f59edba70264cb55533b7cf544982535626f98c\": container with ID starting with 1ea9cf76063beed45e1a77f55f59edba70264cb55533b7cf544982535626f98c not found: ID does not exist" containerID="1ea9cf76063beed45e1a77f55f59edba70264cb55533b7cf544982535626f98c" Oct 30 13:19:19.064336 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.063740 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ea9cf76063beed45e1a77f55f59edba70264cb55533b7cf544982535626f98c"} err="failed to get container status \"1ea9cf76063beed45e1a77f55f59edba70264cb55533b7cf544982535626f98c\": rpc error: code = NotFound desc = could not find container \"1ea9cf76063beed45e1a77f55f59edba70264cb55533b7cf544982535626f98c\": container with ID starting with 1ea9cf76063beed45e1a77f55f59edba70264cb55533b7cf544982535626f98c not found: ID does not exist" Oct 30 13:19:19.096292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.091224 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9" podStartSLOduration=1.71392657 podStartE2EDuration="2.091201455s" totalImagesPullingTime="377.274885ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:19.089040697 +0000 UTC m=+2057.692539145" watchObservedRunningTime="2026-10-30 13:19:19.091201455 +0000 UTC m=+2057.694699903" Oct 30 13:19:19.181912 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.181280 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd" Oct 30 13:19:19.190739 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.188180 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2"] Oct 30 13:19:19.192450 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.192425 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2"] Oct 30 13:19:19.230365 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.229459 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd" podStartSLOduration=2.229438069 podStartE2EDuration="2.229438069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:19.218924305 +0000 UTC m=+2057.822422751" watchObservedRunningTime="2026-10-30 13:19:19.229438069 +0000 UTC m=+2057.832936516" Oct 30 13:19:19.304752 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.304719 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" Oct 30 13:19:19.406547 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.406503 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-workload-certs\") pod \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " Oct 30 13:19:19.406753 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.406581 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-kube-api-access-fk5wq\" (UniqueName: \"kubernetes.io/projected/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-kube-api-access-fk5wq\") pod \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " Oct 30 13:19:19.406753 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.406629 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-istiod-ca-cert\") pod \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " Oct 30 13:19:19.406753 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.406688 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-workload-socket\") pod \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " Oct 30 13:19:19.406919 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.406764 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-credential-socket\") pod \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " Oct 30 13:19:19.406919 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.406801 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-istio-data\") pod \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " Oct 30 13:19:19.406919 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.406862 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-workload-certs" pod "e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6" (UID: "e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:19.406919 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.406881 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-istio-token\" (UniqueName: \"kubernetes.io/projected/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-istio-token\") pod \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " Oct 30 13:19:19.407111 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.406920 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-istio-podinfo\") pod \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " Oct 30 13:19:19.407111 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.407091 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-workload-socket" pod "e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6" (UID: "e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:19.407208 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.407126 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-istio-envoy\") pod \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\" (UID: \"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6\") " Oct 30 13:19:19.407846 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.407389 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-istio-data" pod "e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6" (UID: "e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:19.407846 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.407443 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-istiod-ca-cert" pod "e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6" (UID: "e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:19:19.407846 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.407714 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-credential-socket" pod "e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6" (UID: "e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:19.408077 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.407913 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:19:19.408077 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.407930 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:19:19.408077 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.407944 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:19:19.408077 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.407957 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:19:19.408077 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.407968 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:19:19.411312 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.410670 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-istio-envoy" pod "e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6" (UID: "e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:19.411312 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.411055 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-istio-podinfo" pod "e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6" (UID: "e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:19:19.411312 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.411127 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-kube-api-access-fk5wq" pod "e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6" (UID: "e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6"). InnerVolumeSpecName "kube-api-access-fk5wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:19:19.413153 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.413123 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-istio-token" pod "e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6" (UID: "e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:19:19.509616 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.509509 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:19:19.509616 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.509548 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk5wq\" (UniqueName: \"kubernetes.io/projected/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-kube-api-access-fk5wq\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:19:19.509616 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.509564 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:19:19.509616 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.509580 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:19:19.520424 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:19.520387 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3a9c9f2-eaac-4fcc-88e7-87d311d77677" path="/var/lib/kubelet/pods/d3a9c9f2-eaac-4fcc-88e7-87d311d77677/volumes" Oct 30 13:19:20.063716 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:20.063684 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" Oct 30 13:19:20.063716 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:20.063699 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj" event={"ID":"e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6","Type":"ContainerDied","Data":"39bdf9894fa1bc5f53989243caee664bc894c26c6b9e19347691972cfb8e228e"} Oct 30 13:19:20.064343 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:20.063812 4931 scope.go:122] "RemoveContainer" containerID="d017bb2161ab1ad8e3f37effa68a5f3c5f626dfaeecfba6342eddd0ebda6585e" Oct 30 13:19:20.127554 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:20.127518 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj"] Oct 30 13:19:20.141737 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:20.141695 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj"] Oct 30 13:19:21.515227 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:21.515194 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6" path="/var/lib/kubelet/pods/e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6/volumes" Oct 30 13:19:24.787988 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:24.787949 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8"] Oct 30 13:19:26.083485 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:26.083452 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9" Oct 30 13:19:28.167454 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:28.167420 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd" Oct 30 13:19:28.180553 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:28.180526 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd" Oct 30 13:19:29.170070 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:29.169826 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd" Oct 30 13:19:37.352360 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:19:37.352321 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:19:38.429186 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:38.429148 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm"] Oct 30 13:19:38.429804 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:38.429787 4931 cpu_manager.go:414] "containerMap: removing container" podUID="e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6" containerName="istio-proxy" Oct 30 13:19:38.429847 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:38.429806 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6" containerName="istio-proxy" Oct 30 13:19:38.429847 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:38.429834 4931 cpu_manager.go:414] "containerMap: removing container" podUID="d3a9c9f2-eaac-4fcc-88e7-87d311d77677" containerName="istio-proxy" Oct 30 13:19:38.429847 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:38.429839 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3a9c9f2-eaac-4fcc-88e7-87d311d77677" containerName="istio-proxy" Oct 30 13:19:38.430000 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:38.429944 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5eb3b0b-b1d3-4b9e-83dc-a5179b6c6cb6" containerName="istio-proxy" Oct 30 13:19:38.430000 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:38.429961 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3a9c9f2-eaac-4fcc-88e7-87d311d77677" containerName="istio-proxy" Oct 30 13:19:38.433745 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:38.433723 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm" Oct 30 13:19:38.464797 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:38.464762 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-2b8461-serving-cert\"" Oct 30 13:19:38.471647 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:38.471612 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm"] Oct 30 13:19:38.593217 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:38.593183 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a109d1f-9e17-4638-a451-54d814f34fbd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm\" (UID: \"7a109d1f-9e17-4638-a451-54d814f34fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm" Oct 30 13:19:38.593416 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:38.593377 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a109d1f-9e17-4638-a451-54d814f34fbd-metrics-certs\") pod \"maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm\" (UID: \"7a109d1f-9e17-4638-a451-54d814f34fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm" Oct 30 13:19:38.593528 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:38.593507 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a109d1f-9e17-4638-a451-54d814f34fbd-maas-api-tls\") pod \"maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm\" (UID: \"7a109d1f-9e17-4638-a451-54d814f34fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm" Oct 30 13:19:38.593627 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:38.593607 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjzw8\" (UniqueName: \"kubernetes.io/projected/7a109d1f-9e17-4638-a451-54d814f34fbd-kube-api-access-zjzw8\") pod \"maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm\" (UID: \"7a109d1f-9e17-4638-a451-54d814f34fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm" Oct 30 13:19:38.694629 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:38.694596 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a109d1f-9e17-4638-a451-54d814f34fbd-metrics-certs\") pod \"maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm\" (UID: \"7a109d1f-9e17-4638-a451-54d814f34fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm" Oct 30 13:19:38.694810 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:38.694699 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a109d1f-9e17-4638-a451-54d814f34fbd-maas-api-tls\") pod \"maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm\" (UID: \"7a109d1f-9e17-4638-a451-54d814f34fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm" Oct 30 13:19:38.694880 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:38.694806 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjzw8\" (UniqueName: \"kubernetes.io/projected/7a109d1f-9e17-4638-a451-54d814f34fbd-kube-api-access-zjzw8\") pod \"maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm\" (UID: \"7a109d1f-9e17-4638-a451-54d814f34fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm" Oct 30 13:19:38.694949 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:38.694886 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a109d1f-9e17-4638-a451-54d814f34fbd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm\" (UID: \"7a109d1f-9e17-4638-a451-54d814f34fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm" Oct 30 13:19:38.695706 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:38.695679 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a109d1f-9e17-4638-a451-54d814f34fbd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm\" (UID: \"7a109d1f-9e17-4638-a451-54d814f34fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm" Oct 30 13:19:38.713319 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:38.702864 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a109d1f-9e17-4638-a451-54d814f34fbd-metrics-certs\") pod \"maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm\" (UID: \"7a109d1f-9e17-4638-a451-54d814f34fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm" Oct 30 13:19:38.721522 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:38.714144 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a109d1f-9e17-4638-a451-54d814f34fbd-maas-api-tls\") pod \"maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm\" (UID: \"7a109d1f-9e17-4638-a451-54d814f34fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm" Oct 30 13:19:38.732344 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:38.731480 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjzw8\" (UniqueName: \"kubernetes.io/projected/7a109d1f-9e17-4638-a451-54d814f34fbd-kube-api-access-zjzw8\") pod \"maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm\" (UID: \"7a109d1f-9e17-4638-a451-54d814f34fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm" Oct 30 13:19:38.747879 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:38.747845 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm" Oct 30 13:19:39.072799 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:39.072774 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm"] Oct 30 13:19:39.073359 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:19:39.073321 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a109d1f_9e17_4638_a451_54d814f34fbd.slice/crio-8fcf892992db13d197166771faa810f2062cd94630633cea61f69cc1874e795e WatchSource:0}: Error finding container 8fcf892992db13d197166771faa810f2062cd94630633cea61f69cc1874e795e: Status 404 returned error can't find the container with id 8fcf892992db13d197166771faa810f2062cd94630633cea61f69cc1874e795e Oct 30 13:19:39.228662 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:39.228627 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm" event={"ID":"7a109d1f-9e17-4638-a451-54d814f34fbd","Type":"ContainerStarted","Data":"8fcf892992db13d197166771faa810f2062cd94630633cea61f69cc1874e795e"} Oct 30 13:19:39.813997 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:39.813913 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8" podUID="844f8313-ec50-42d9-9f6d-7a0467b7a95b" containerName="main" containerID="cri-o://96403e9d198cce9795668c3885edd7f8ccb1b9f516ae69126d6ad46098efd1f0" gracePeriod=45 Oct 30 13:19:40.095201 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:40.095175 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8" Oct 30 13:19:40.218151 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:40.218118 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/844f8313-ec50-42d9-9f6d-7a0467b7a95b-kube-api-access-9lxfj\" (UniqueName: \"kubernetes.io/projected/844f8313-ec50-42d9-9f6d-7a0467b7a95b-kube-api-access-9lxfj\") pod \"844f8313-ec50-42d9-9f6d-7a0467b7a95b\" (UID: \"844f8313-ec50-42d9-9f6d-7a0467b7a95b\") " Oct 30 13:19:40.218392 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:40.218248 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/844f8313-ec50-42d9-9f6d-7a0467b7a95b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/844f8313-ec50-42d9-9f6d-7a0467b7a95b-tmp-dir\") pod \"844f8313-ec50-42d9-9f6d-7a0467b7a95b\" (UID: \"844f8313-ec50-42d9-9f6d-7a0467b7a95b\") " Oct 30 13:19:40.218392 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:40.218298 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/844f8313-ec50-42d9-9f6d-7a0467b7a95b-tls-certs\" (UniqueName: \"kubernetes.io/secret/844f8313-ec50-42d9-9f6d-7a0467b7a95b-tls-certs\") pod \"844f8313-ec50-42d9-9f6d-7a0467b7a95b\" (UID: \"844f8313-ec50-42d9-9f6d-7a0467b7a95b\") " Oct 30 13:19:40.218392 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:40.218351 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/844f8313-ec50-42d9-9f6d-7a0467b7a95b-home\" (UniqueName: \"kubernetes.io/empty-dir/844f8313-ec50-42d9-9f6d-7a0467b7a95b-home\") pod \"844f8313-ec50-42d9-9f6d-7a0467b7a95b\" (UID: \"844f8313-ec50-42d9-9f6d-7a0467b7a95b\") " Oct 30 13:19:40.218587 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:40.218464 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/844f8313-ec50-42d9-9f6d-7a0467b7a95b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/844f8313-ec50-42d9-9f6d-7a0467b7a95b-dshm\") pod \"844f8313-ec50-42d9-9f6d-7a0467b7a95b\" (UID: \"844f8313-ec50-42d9-9f6d-7a0467b7a95b\") " Oct 30 13:19:40.218587 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:40.218516 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/844f8313-ec50-42d9-9f6d-7a0467b7a95b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/844f8313-ec50-42d9-9f6d-7a0467b7a95b-model-cache\") pod \"844f8313-ec50-42d9-9f6d-7a0467b7a95b\" (UID: \"844f8313-ec50-42d9-9f6d-7a0467b7a95b\") " Oct 30 13:19:40.218587 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:40.218566 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/844f8313-ec50-42d9-9f6d-7a0467b7a95b-tmp-dir" pod "844f8313-ec50-42d9-9f6d-7a0467b7a95b" (UID: "844f8313-ec50-42d9-9f6d-7a0467b7a95b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:40.218772 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:40.218683 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/844f8313-ec50-42d9-9f6d-7a0467b7a95b-home" pod "844f8313-ec50-42d9-9f6d-7a0467b7a95b" (UID: "844f8313-ec50-42d9-9f6d-7a0467b7a95b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:40.218830 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:40.218805 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/844f8313-ec50-42d9-9f6d-7a0467b7a95b-model-cache" pod "844f8313-ec50-42d9-9f6d-7a0467b7a95b" (UID: "844f8313-ec50-42d9-9f6d-7a0467b7a95b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:40.219214 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:40.219191 4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/844f8313-ec50-42d9-9f6d-7a0467b7a95b-tmp-dir\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:19:40.219214 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:40.219213 4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/844f8313-ec50-42d9-9f6d-7a0467b7a95b-home\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:19:40.219440 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:40.219222 4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/844f8313-ec50-42d9-9f6d-7a0467b7a95b-model-cache\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:19:40.220654 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:40.220629 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/844f8313-ec50-42d9-9f6d-7a0467b7a95b-dshm" pod "844f8313-ec50-42d9-9f6d-7a0467b7a95b" (UID: "844f8313-ec50-42d9-9f6d-7a0467b7a95b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:40.220774 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:40.220757 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/844f8313-ec50-42d9-9f6d-7a0467b7a95b-tls-certs" pod "844f8313-ec50-42d9-9f6d-7a0467b7a95b" (UID: "844f8313-ec50-42d9-9f6d-7a0467b7a95b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:19:40.220890 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:40.220870 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/844f8313-ec50-42d9-9f6d-7a0467b7a95b-kube-api-access-9lxfj" pod "844f8313-ec50-42d9-9f6d-7a0467b7a95b" (UID: "844f8313-ec50-42d9-9f6d-7a0467b7a95b"). InnerVolumeSpecName "kube-api-access-9lxfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:19:40.239556 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:40.239523 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm" event={"ID":"7a109d1f-9e17-4638-a451-54d814f34fbd","Type":"ContainerStarted","Data":"9b0ed253e5e5eda6f0f0db8bf81d4f8e2467e85f8e5a4732e10abc3f19d40320"} Oct 30 13:19:40.239673 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:40.239594 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm" Oct 30 13:19:40.240927 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:40.240907 4931 generic.go:356] "Generic (PLEG): container finished" podID="844f8313-ec50-42d9-9f6d-7a0467b7a95b" containerID="96403e9d198cce9795668c3885edd7f8ccb1b9f516ae69126d6ad46098efd1f0" exitCode=0 Oct 30 13:19:40.241002 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:40.240988 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8" Oct 30 13:19:40.241064 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:40.241039 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8" event={"ID":"844f8313-ec50-42d9-9f6d-7a0467b7a95b","Type":"ContainerDied","Data":"96403e9d198cce9795668c3885edd7f8ccb1b9f516ae69126d6ad46098efd1f0"} Oct 30 13:19:40.241127 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:40.241088 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8" event={"ID":"844f8313-ec50-42d9-9f6d-7a0467b7a95b","Type":"ContainerDied","Data":"f513088405425286b33e3dc6b2405c30a69a5f06e4d54cf465e54d56b9a48f15"} Oct 30 13:19:40.241127 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:40.241113 4931 scope.go:122] "RemoveContainer" containerID="96403e9d198cce9795668c3885edd7f8ccb1b9f516ae69126d6ad46098efd1f0" Oct 30 13:19:40.251577 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:40.251559 4931 scope.go:122] "RemoveContainer" containerID="96403e9d198cce9795668c3885edd7f8ccb1b9f516ae69126d6ad46098efd1f0" Oct 30 13:19:40.251819 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:19:40.251800 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96403e9d198cce9795668c3885edd7f8ccb1b9f516ae69126d6ad46098efd1f0\": container with ID starting with 96403e9d198cce9795668c3885edd7f8ccb1b9f516ae69126d6ad46098efd1f0 not found: ID does not exist" containerID="96403e9d198cce9795668c3885edd7f8ccb1b9f516ae69126d6ad46098efd1f0" Oct 30 13:19:40.251878 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:40.251825 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96403e9d198cce9795668c3885edd7f8ccb1b9f516ae69126d6ad46098efd1f0"} err="failed to get container status \"96403e9d198cce9795668c3885edd7f8ccb1b9f516ae69126d6ad46098efd1f0\": rpc error: code = NotFound desc = could not find container \"96403e9d198cce9795668c3885edd7f8ccb1b9f516ae69126d6ad46098efd1f0\": container with ID starting with 96403e9d198cce9795668c3885edd7f8ccb1b9f516ae69126d6ad46098efd1f0 not found: ID does not exist" Oct 30 13:19:40.320606 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:40.320569 4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/844f8313-ec50-42d9-9f6d-7a0467b7a95b-tls-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:19:40.320606 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:40.320598 4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/844f8313-ec50-42d9-9f6d-7a0467b7a95b-dshm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:19:40.320767 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:40.320613 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lxfj\" (UniqueName: \"kubernetes.io/projected/844f8313-ec50-42d9-9f6d-7a0467b7a95b-kube-api-access-9lxfj\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:19:40.416353 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:40.416251 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm" podStartSLOduration=2.028106494 podStartE2EDuration="2.41623799s" totalImagesPullingTime="388.131496ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:40.336227414 +0000 UTC m=+2078.939725860" watchObservedRunningTime="2026-10-30 13:19:40.41623799 +0000 UTC m=+2079.019736436" Oct 30 13:19:40.417623 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:40.417602 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8"] Oct 30 13:19:40.424596 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:40.424570 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-2a3e01/test-model-2a3e01-kserve-54dbfc6f9-c84l8"] Oct 30 13:19:41.515784 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:41.515747 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="844f8313-ec50-42d9-9f6d-7a0467b7a95b" path="/var/lib/kubelet/pods/844f8313-ec50-42d9-9f6d-7a0467b7a95b/volumes" Oct 30 13:19:46.255640 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:46.255609 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm" Oct 30 13:19:52.044968 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:52.044926 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70ba7ee1-v2nlk"] Oct 30 13:19:52.055967 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:52.055933 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70ba7ee1-v2nlk"] Oct 30 13:19:53.515668 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:53.515633 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc5c16ca-a37d-41c0-8bd4-246a23d03189" path="/var/lib/kubelet/pods/dc5c16ca-a37d-41c0-8bd4-246a23d03189/volumes" Oct 30 13:19:55.250398 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:55.250359 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb"] Oct 30 13:19:55.251261 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:55.251231 4931 cpu_manager.go:414] "containerMap: removing container" podUID="844f8313-ec50-42d9-9f6d-7a0467b7a95b" containerName="main" Oct 30 13:19:55.251261 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:55.251257 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="844f8313-ec50-42d9-9f6d-7a0467b7a95b" containerName="main" Oct 30 13:19:55.251575 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:55.251422 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="844f8313-ec50-42d9-9f6d-7a0467b7a95b" containerName="main" Oct 30 13:19:55.255835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:55.255812 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb" Oct 30 13:19:55.282659 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:55.282622 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb"] Oct 30 13:19:55.291837 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:55.291803 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-80fbd9\"/\"kube-root-ca.crt\"" Oct 30 13:19:55.292021 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:55.291861 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-80fbd9\"/\"default-dockercfg-jlfx6\"" Oct 30 13:19:55.296772 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:55.296715 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-80fbd9\"/\"openshift-service-ca.crt\"" Oct 30 13:19:55.296957 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:55.296804 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-80fbd9\"/\"test-model-80fbd9-kserve-self-signed-certs\"" Oct 30 13:19:55.351677 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:55.351646 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c92125c-50d9-4d98-9978-32c7c94eab42-home\") pod \"test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb\" (UID: \"2c92125c-50d9-4d98-9978-32c7c94eab42\") " pod="ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb" Oct 30 13:19:55.351894 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:55.351788 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c92125c-50d9-4d98-9978-32c7c94eab42-dshm\") pod \"test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb\" (UID: \"2c92125c-50d9-4d98-9978-32c7c94eab42\") " pod="ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb" Oct 30 13:19:55.351971 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:55.351931 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c92125c-50d9-4d98-9978-32c7c94eab42-tls-certs\") pod \"test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb\" (UID: \"2c92125c-50d9-4d98-9978-32c7c94eab42\") " pod="ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb" Oct 30 13:19:55.352035 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:55.352016 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c92125c-50d9-4d98-9978-32c7c94eab42-tmp-dir\") pod \"test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb\" (UID: \"2c92125c-50d9-4d98-9978-32c7c94eab42\") " pod="ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb" Oct 30 13:19:55.352141 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:55.352125 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdjjw\" (UniqueName: \"kubernetes.io/projected/2c92125c-50d9-4d98-9978-32c7c94eab42-kube-api-access-bdjjw\") pod \"test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb\" (UID: \"2c92125c-50d9-4d98-9978-32c7c94eab42\") " pod="ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb" Oct 30 13:19:55.352244 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:55.352221 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c92125c-50d9-4d98-9978-32c7c94eab42-model-cache\") pod \"test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb\" (UID: \"2c92125c-50d9-4d98-9978-32c7c94eab42\") " pod="ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb" Oct 30 13:19:55.453885 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:55.453844 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c92125c-50d9-4d98-9978-32c7c94eab42-home\") pod \"test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb\" (UID: \"2c92125c-50d9-4d98-9978-32c7c94eab42\") " pod="ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb" Oct 30 13:19:55.454096 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:55.453947 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c92125c-50d9-4d98-9978-32c7c94eab42-dshm\") pod \"test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb\" (UID: \"2c92125c-50d9-4d98-9978-32c7c94eab42\") " pod="ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb" Oct 30 13:19:55.454096 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:55.454037 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c92125c-50d9-4d98-9978-32c7c94eab42-tls-certs\") pod \"test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb\" (UID: \"2c92125c-50d9-4d98-9978-32c7c94eab42\") " pod="ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb" Oct 30 13:19:55.454096 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:55.454089 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c92125c-50d9-4d98-9978-32c7c94eab42-tmp-dir\") pod \"test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb\" (UID: \"2c92125c-50d9-4d98-9978-32c7c94eab42\") " pod="ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb" Oct 30 13:19:55.454262 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:55.454166 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdjjw\" (UniqueName: \"kubernetes.io/projected/2c92125c-50d9-4d98-9978-32c7c94eab42-kube-api-access-bdjjw\") pod \"test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb\" (UID: \"2c92125c-50d9-4d98-9978-32c7c94eab42\") " pod="ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb" Oct 30 13:19:55.454262 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:55.454212 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c92125c-50d9-4d98-9978-32c7c94eab42-model-cache\") pod \"test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb\" (UID: \"2c92125c-50d9-4d98-9978-32c7c94eab42\") " pod="ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb" Oct 30 13:19:55.454422 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:55.454395 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c92125c-50d9-4d98-9978-32c7c94eab42-home\") pod \"test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb\" (UID: \"2c92125c-50d9-4d98-9978-32c7c94eab42\") " pod="ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb" Oct 30 13:19:55.454546 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:55.454523 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c92125c-50d9-4d98-9978-32c7c94eab42-tmp-dir\") pod \"test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb\" (UID: \"2c92125c-50d9-4d98-9978-32c7c94eab42\") " pod="ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb" Oct 30 13:19:55.454626 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:55.454588 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c92125c-50d9-4d98-9978-32c7c94eab42-model-cache\") pod \"test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb\" (UID: \"2c92125c-50d9-4d98-9978-32c7c94eab42\") " pod="ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb" Oct 30 13:19:55.456801 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:55.456770 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c92125c-50d9-4d98-9978-32c7c94eab42-dshm\") pod \"test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb\" (UID: \"2c92125c-50d9-4d98-9978-32c7c94eab42\") " pod="ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb" Oct 30 13:19:55.456939 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:55.456828 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c92125c-50d9-4d98-9978-32c7c94eab42-tls-certs\") pod \"test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb\" (UID: \"2c92125c-50d9-4d98-9978-32c7c94eab42\") " pod="ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb" Oct 30 13:19:55.490150 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:55.490112 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdjjw\" (UniqueName: \"kubernetes.io/projected/2c92125c-50d9-4d98-9978-32c7c94eab42-kube-api-access-bdjjw\") pod \"test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb\" (UID: \"2c92125c-50d9-4d98-9978-32c7c94eab42\") " pod="ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb" Oct 30 13:19:55.550131 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:55.550031 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:19:55.574694 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:55.574657 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb" Oct 30 13:19:56.021172 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:19:56.021134 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c92125c_50d9_4d98_9978_32c7c94eab42.slice/crio-bb00b8e0d05a02c47bb78044d3936a3e0927457f34251a53299195829c246bd2 WatchSource:0}: Error finding container bb00b8e0d05a02c47bb78044d3936a3e0927457f34251a53299195829c246bd2: Status 404 returned error can't find the container with id bb00b8e0d05a02c47bb78044d3936a3e0927457f34251a53299195829c246bd2 Oct 30 13:19:56.029589 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:56.029556 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb"] Oct 30 13:19:56.366982 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:56.366894 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb" event={"ID":"2c92125c-50d9-4d98-9978-32c7c94eab42","Type":"ContainerStarted","Data":"9ba6972a468ecc349a21d58d857071d17571bf2de533a3297a72f3564b6114ed"} Oct 30 13:19:56.366982 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:56.366942 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb" event={"ID":"2c92125c-50d9-4d98-9978-32c7c94eab42","Type":"ContainerStarted","Data":"bb00b8e0d05a02c47bb78044d3936a3e0927457f34251a53299195829c246bd2"} Oct 30 13:19:56.420287 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:56.420222 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb" podStartSLOduration=1.42020667 podStartE2EDuration="1.42020667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:56.419398995 +0000 UTC m=+2095.022897443" watchObservedRunningTime="2026-10-30 13:19:56.42020667 +0000 UTC m=+2095.023705113" Oct 30 13:19:56.575973 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:56.575938 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb" Oct 30 13:19:58.372629 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:58.372107 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc"] Oct 30 13:19:59.205101 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:59.205061 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv"] Oct 30 13:19:59.209503 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:59.209484 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv" Oct 30 13:19:59.235490 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:59.235449 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-619b9b-serving-cert\"" Oct 30 13:19:59.254991 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:59.254959 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv"] Oct 30 13:19:59.280594 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:59.280543 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkk5h\" (UniqueName: \"kubernetes.io/projected/441dfaf9-d63b-4882-a70b-f1769a32ca9d-kube-api-access-wkk5h\") pod \"maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv\" (UID: \"441dfaf9-d63b-4882-a70b-f1769a32ca9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv" Oct 30 13:19:59.280811 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:59.280790 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/441dfaf9-d63b-4882-a70b-f1769a32ca9d-metrics-certs\") pod \"maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv\" (UID: \"441dfaf9-d63b-4882-a70b-f1769a32ca9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv" Oct 30 13:19:59.281032 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:59.281006 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/441dfaf9-d63b-4882-a70b-f1769a32ca9d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv\" (UID: \"441dfaf9-d63b-4882-a70b-f1769a32ca9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv" Oct 30 13:19:59.281158 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:59.281134 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/441dfaf9-d63b-4882-a70b-f1769a32ca9d-maas-api-tls\") pod \"maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv\" (UID: \"441dfaf9-d63b-4882-a70b-f1769a32ca9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv" Oct 30 13:19:59.382679 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:59.382632 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/441dfaf9-d63b-4882-a70b-f1769a32ca9d-metrics-certs\") pod \"maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv\" (UID: \"441dfaf9-d63b-4882-a70b-f1769a32ca9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv" Oct 30 13:19:59.383233 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:59.382873 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/441dfaf9-d63b-4882-a70b-f1769a32ca9d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv\" (UID: \"441dfaf9-d63b-4882-a70b-f1769a32ca9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv" Oct 30 13:19:59.383233 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:59.382961 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/441dfaf9-d63b-4882-a70b-f1769a32ca9d-maas-api-tls\") pod \"maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv\" (UID: \"441dfaf9-d63b-4882-a70b-f1769a32ca9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv" Oct 30 13:19:59.383233 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:59.383025 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkk5h\" (UniqueName: \"kubernetes.io/projected/441dfaf9-d63b-4882-a70b-f1769a32ca9d-kube-api-access-wkk5h\") pod \"maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv\" (UID: \"441dfaf9-d63b-4882-a70b-f1769a32ca9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv" Oct 30 13:19:59.384183 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:59.384158 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/441dfaf9-d63b-4882-a70b-f1769a32ca9d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv\" (UID: \"441dfaf9-d63b-4882-a70b-f1769a32ca9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv" Oct 30 13:19:59.386020 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:59.385998 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/441dfaf9-d63b-4882-a70b-f1769a32ca9d-metrics-certs\") pod \"maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv\" (UID: \"441dfaf9-d63b-4882-a70b-f1769a32ca9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv" Oct 30 13:19:59.386217 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:59.386196 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/441dfaf9-d63b-4882-a70b-f1769a32ca9d-maas-api-tls\") pod \"maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv\" (UID: \"441dfaf9-d63b-4882-a70b-f1769a32ca9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv" Oct 30 13:19:59.427742 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:59.427697 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkk5h\" (UniqueName: \"kubernetes.io/projected/441dfaf9-d63b-4882-a70b-f1769a32ca9d-kube-api-access-wkk5h\") pod \"maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv\" (UID: \"441dfaf9-d63b-4882-a70b-f1769a32ca9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv" Oct 30 13:19:59.525823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:59.525743 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv" Oct 30 13:19:59.862887 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:59.862853 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv"] Oct 30 13:19:59.865945 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:19:59.865913 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod441dfaf9_d63b_4882_a70b_f1769a32ca9d.slice/crio-9b0729780b8deeb8e16b5581cc4a740586423581bcfa5ec800632e25b32f0112 WatchSource:0}: Error finding container 9b0729780b8deeb8e16b5581cc4a740586423581bcfa5ec800632e25b32f0112: Status 404 returned error can't find the container with id 9b0729780b8deeb8e16b5581cc4a740586423581bcfa5ec800632e25b32f0112 Oct 30 13:19:59.867820 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:19:59.867795 4931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:20:00.414101 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:00.414056 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv" event={"ID":"441dfaf9-d63b-4882-a70b-f1769a32ca9d","Type":"ContainerStarted","Data":"312305c9682e71a54ccd183a2455f6906ac79748c0b0645f794192b271291920"} Oct 30 13:20:00.414674 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:00.414128 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv" event={"ID":"441dfaf9-d63b-4882-a70b-f1769a32ca9d","Type":"ContainerStarted","Data":"9b0729780b8deeb8e16b5581cc4a740586423581bcfa5ec800632e25b32f0112"} Oct 30 13:20:00.414674 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:00.414288 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv" Oct 30 13:20:00.502679 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:00.502625 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv" podStartSLOduration=1.116360545 podStartE2EDuration="1.502607677s" totalImagesPullingTime="386.247132ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:00.500709388 +0000 UTC m=+2099.104207834" watchObservedRunningTime="2026-10-30 13:20:00.502607677 +0000 UTC m=+2099.106106124" Oct 30 13:20:01.928749 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:01.928718 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/2.log" Oct 30 13:20:01.937850 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:01.937820 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/2.log" Oct 30 13:20:02.739204 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:02.739167 4931 scope.go:122] "RemoveContainer" containerID="d2937ccb41823d953519e3a1272f0ba27ba2cdda6fb163e7131f1e7e3e80a4df" Oct 30 13:20:02.752247 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:02.752217 4931 scope.go:122] "RemoveContainer" containerID="f38acf38fe3a892125cc703a1ac57fb4e74fcaf42cbd4b9cf6ef27f55ea5dfca" Oct 30 13:20:02.762924 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:02.762903 4931 scope.go:122] "RemoveContainer" containerID="3be1ed0eccb9cc0943b0eba3ceae7d4dbace4d8976a3e731a5edf51e10c39218" Oct 30 13:20:05.575480 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:05.575439 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb" Oct 30 13:20:05.588699 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:05.588660 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb" Oct 30 13:20:06.430054 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:06.430022 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv" Oct 30 13:20:06.539891 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:06.539854 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb" Oct 30 13:20:07.353507 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:20:07.353470 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:20:13.182097 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:13.182006 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz"] Oct 30 13:20:13.400593 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:13.400556 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc" podUID="823d8e4a-010a-4760-a070-f05bc6c2d7cc" containerName="main" containerID="cri-o://6894ab6f9fb6b26049b84e9844a9e86ea7dcd86a36d74f51885da3c1f3103341" gracePeriod=45 Oct 30 13:20:13.587453 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:13.587426 4931 generic.go:356] "Generic (PLEG): container finished" podID="823d8e4a-010a-4760-a070-f05bc6c2d7cc" containerID="6894ab6f9fb6b26049b84e9844a9e86ea7dcd86a36d74f51885da3c1f3103341" exitCode=0 Oct 30 13:20:13.587582 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:13.587540 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc" event={"ID":"823d8e4a-010a-4760-a070-f05bc6c2d7cc","Type":"ContainerDied","Data":"6894ab6f9fb6b26049b84e9844a9e86ea7dcd86a36d74f51885da3c1f3103341"} Oct 30 13:20:13.701316 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:13.701288 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc" Oct 30 13:20:13.805998 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:13.805913 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823d8e4a-010a-4760-a070-f05bc6c2d7cc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/823d8e4a-010a-4760-a070-f05bc6c2d7cc-dshm\") pod \"823d8e4a-010a-4760-a070-f05bc6c2d7cc\" (UID: \"823d8e4a-010a-4760-a070-f05bc6c2d7cc\") " Oct 30 13:20:13.806148 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:13.806005 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823d8e4a-010a-4760-a070-f05bc6c2d7cc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/823d8e4a-010a-4760-a070-f05bc6c2d7cc-tmp-dir\") pod \"823d8e4a-010a-4760-a070-f05bc6c2d7cc\" (UID: \"823d8e4a-010a-4760-a070-f05bc6c2d7cc\") " Oct 30 13:20:13.806148 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:13.806033 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823d8e4a-010a-4760-a070-f05bc6c2d7cc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/823d8e4a-010a-4760-a070-f05bc6c2d7cc-model-cache\") pod \"823d8e4a-010a-4760-a070-f05bc6c2d7cc\" (UID: \"823d8e4a-010a-4760-a070-f05bc6c2d7cc\") " Oct 30 13:20:13.806148 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:13.806069 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823d8e4a-010a-4760-a070-f05bc6c2d7cc-home\" (UniqueName: \"kubernetes.io/empty-dir/823d8e4a-010a-4760-a070-f05bc6c2d7cc-home\") pod \"823d8e4a-010a-4760-a070-f05bc6c2d7cc\" (UID: \"823d8e4a-010a-4760-a070-f05bc6c2d7cc\") " Oct 30 13:20:13.806390 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:13.806240 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/823d8e4a-010a-4760-a070-f05bc6c2d7cc-tls-certs\" (UniqueName: \"kubernetes.io/secret/823d8e4a-010a-4760-a070-f05bc6c2d7cc-tls-certs\") pod \"823d8e4a-010a-4760-a070-f05bc6c2d7cc\" (UID: \"823d8e4a-010a-4760-a070-f05bc6c2d7cc\") " Oct 30 13:20:13.806390 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:13.806339 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823d8e4a-010a-4760-a070-f05bc6c2d7cc-tmp-dir" pod "823d8e4a-010a-4760-a070-f05bc6c2d7cc" (UID: "823d8e4a-010a-4760-a070-f05bc6c2d7cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:13.806390 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:13.806347 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823d8e4a-010a-4760-a070-f05bc6c2d7cc-model-cache" pod "823d8e4a-010a-4760-a070-f05bc6c2d7cc" (UID: "823d8e4a-010a-4760-a070-f05bc6c2d7cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:13.806546 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:13.806411 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/823d8e4a-010a-4760-a070-f05bc6c2d7cc-kube-api-access-qqpfw\" (UniqueName: \"kubernetes.io/projected/823d8e4a-010a-4760-a070-f05bc6c2d7cc-kube-api-access-qqpfw\") pod \"823d8e4a-010a-4760-a070-f05bc6c2d7cc\" (UID: \"823d8e4a-010a-4760-a070-f05bc6c2d7cc\") " Oct 30 13:20:13.806546 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:13.806436 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823d8e4a-010a-4760-a070-f05bc6c2d7cc-home" pod "823d8e4a-010a-4760-a070-f05bc6c2d7cc" (UID: "823d8e4a-010a-4760-a070-f05bc6c2d7cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:13.807019 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:13.806991 4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/823d8e4a-010a-4760-a070-f05bc6c2d7cc-home\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:20:13.807019 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:13.807018 4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/823d8e4a-010a-4760-a070-f05bc6c2d7cc-tmp-dir\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:20:13.807220 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:13.807035 4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/823d8e4a-010a-4760-a070-f05bc6c2d7cc-model-cache\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:20:13.808695 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:13.808665 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823d8e4a-010a-4760-a070-f05bc6c2d7cc-dshm" pod "823d8e4a-010a-4760-a070-f05bc6c2d7cc" (UID: "823d8e4a-010a-4760-a070-f05bc6c2d7cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:13.808799 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:13.808755 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/823d8e4a-010a-4760-a070-f05bc6c2d7cc-tls-certs" pod "823d8e4a-010a-4760-a070-f05bc6c2d7cc" (UID: "823d8e4a-010a-4760-a070-f05bc6c2d7cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:20:13.808864 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:13.808822 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823d8e4a-010a-4760-a070-f05bc6c2d7cc-kube-api-access-qqpfw" pod "823d8e4a-010a-4760-a070-f05bc6c2d7cc" (UID: "823d8e4a-010a-4760-a070-f05bc6c2d7cc"). InnerVolumeSpecName "kube-api-access-qqpfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:20:13.908810 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:13.908777 4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/823d8e4a-010a-4760-a070-f05bc6c2d7cc-tls-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:20:13.908810 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:13.908806 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqpfw\" (UniqueName: \"kubernetes.io/projected/823d8e4a-010a-4760-a070-f05bc6c2d7cc-kube-api-access-qqpfw\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:20:13.908810 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:13.908817 4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/823d8e4a-010a-4760-a070-f05bc6c2d7cc-dshm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:20:14.596490 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:14.596455 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc" event={"ID":"823d8e4a-010a-4760-a070-f05bc6c2d7cc","Type":"ContainerDied","Data":"16213195f102cd87ea31bee6642cf466748412eff091fdb57c6c69db1a18b2fc"} Oct 30 13:20:14.596490 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:14.596476 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc" Oct 30 13:20:14.597244 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:14.596502 4931 scope.go:122] "RemoveContainer" containerID="6894ab6f9fb6b26049b84e9844a9e86ea7dcd86a36d74f51885da3c1f3103341" Oct 30 13:20:14.682017 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:14.681970 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc"] Oct 30 13:20:14.691515 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:14.691487 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-649303/test-model-649303-kserve-6f78cffb4f-hgxxc"] Oct 30 13:20:15.441296 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.441249 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz"] Oct 30 13:20:15.441893 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.441879 4931 cpu_manager.go:414] "containerMap: removing container" podUID="823d8e4a-010a-4760-a070-f05bc6c2d7cc" containerName="main" Oct 30 13:20:15.441947 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.441895 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="823d8e4a-010a-4760-a070-f05bc6c2d7cc" containerName="main" Oct 30 13:20:15.442035 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.442023 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="823d8e4a-010a-4760-a070-f05bc6c2d7cc" containerName="main" Oct 30 13:20:15.446840 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.446817 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" Oct 30 13:20:15.464073 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.464045 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6af611-openshift-default-dockercfg-s4h66\"" Oct 30 13:20:15.481767 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.481711 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz"] Oct 30 13:20:15.518987 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.518943 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="823d8e4a-010a-4760-a070-f05bc6c2d7cc" path="/var/lib/kubelet/pods/823d8e4a-010a-4760-a070-f05bc6c2d7cc/volumes" Oct 30 13:20:15.536775 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.536742 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-istio-token\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" Oct 30 13:20:15.536960 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.536787 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-workload-certs\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" Oct 30 13:20:15.536960 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.536806 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-istiod-ca-cert\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" Oct 30 13:20:15.537086 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.536970 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwv58\" (UniqueName: \"kubernetes.io/projected/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-kube-api-access-mwv58\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" Oct 30 13:20:15.537086 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.537030 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-workload-socket\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" Oct 30 13:20:15.537311 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.537112 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-credential-socket\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" Oct 30 13:20:15.537311 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.537245 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-istio-podinfo\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" Oct 30 13:20:15.537444 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.537335 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-istio-data\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" Oct 30 13:20:15.537444 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.537375 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-istio-envoy\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" Oct 30 13:20:15.638604 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.638564 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-istio-podinfo\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" Oct 30 13:20:15.638604 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.638612 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-istio-data\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" Oct 30 13:20:15.639129 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.638785 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-istio-envoy\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" Oct 30 13:20:15.639129 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.638970 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-istio-token\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" Oct 30 13:20:15.639129 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.639013 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-istio-data\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" Oct 30 13:20:15.639129 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.639019 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-workload-certs\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" Oct 30 13:20:15.639129 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.639076 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-istiod-ca-cert\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" Oct 30 13:20:15.639401 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.639145 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwv58\" (UniqueName: \"kubernetes.io/projected/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-kube-api-access-mwv58\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" Oct 30 13:20:15.639401 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.639175 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-workload-socket\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" Oct 30 13:20:15.639401 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.639204 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-credential-socket\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" Oct 30 13:20:15.639401 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.639324 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-workload-certs\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" Oct 30 13:20:15.639617 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.639596 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-workload-socket\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" Oct 30 13:20:15.639705 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.639687 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-credential-socket\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" Oct 30 13:20:15.640076 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.640052 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-istiod-ca-cert\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" Oct 30 13:20:15.641110 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.641086 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-istio-envoy\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" Oct 30 13:20:15.641408 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.641388 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-istio-podinfo\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" Oct 30 13:20:15.665884 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.665853 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-istio-token\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" Oct 30 13:20:15.666010 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.665984 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwv58\" (UniqueName: \"kubernetes.io/projected/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-kube-api-access-mwv58\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" Oct 30 13:20:15.761490 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:15.761454 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" Oct 30 13:20:16.089279 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:20:16.089236 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod608feda4_9f34_43d8_b6bb_e3cc096f7d5c.slice/crio-b169389fa9fdae0d2d92c67026ff80d0e96150f40d924de16b90b2591a56a85a WatchSource:0}: Error finding container b169389fa9fdae0d2d92c67026ff80d0e96150f40d924de16b90b2591a56a85a: Status 404 returned error can't find the container with id b169389fa9fdae0d2d92c67026ff80d0e96150f40d924de16b90b2591a56a85a Oct 30 13:20:16.090259 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:16.090238 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz"] Oct 30 13:20:16.092245 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:16.092213 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:20:16.092367 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:16.092291 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:20:16.092367 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:16.092323 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:20:16.614550 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:16.614510 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" event={"ID":"608feda4-9f34-43d8-b6bb-e3cc096f7d5c","Type":"ContainerStarted","Data":"8ec474d83771971ce7039d5b54d59419a111363f2e5b4b91942502c49dcfc705"} Oct 30 13:20:16.614759 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:16.614578 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" event={"ID":"608feda4-9f34-43d8-b6bb-e3cc096f7d5c","Type":"ContainerStarted","Data":"b169389fa9fdae0d2d92c67026ff80d0e96150f40d924de16b90b2591a56a85a"} Oct 30 13:20:16.714287 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:16.711412 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" podStartSLOduration=1.711388759 podStartE2EDuration="1.711388759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:16.70474845 +0000 UTC m=+2115.308246910" watchObservedRunningTime="2026-10-30 13:20:16.711388759 +0000 UTC m=+2115.314887206" Oct 30 13:20:16.764492 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:16.764408 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" Oct 30 13:20:17.764764 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:17.764718 4931 patch_prober.go:29] interesting pod/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 13:20:17.765304 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:17.764782 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" podUID="608feda4-9f34-43d8-b6bb-e3cc096f7d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 13:20:18.775356 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:18.765559 4931 patch_prober.go:29] interesting pod/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 13:20:18.775356 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:18.765631 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" podUID="608feda4-9f34-43d8-b6bb-e3cc096f7d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 13:20:19.762812 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:19.762774 4931 patch_prober.go:29] interesting pod/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 13:20:19.763020 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:19.762830 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" podUID="608feda4-9f34-43d8-b6bb-e3cc096f7d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 13:20:20.762149 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:20.762110 4931 patch_prober.go:29] interesting pod/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 13:20:20.762664 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:20.762173 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" podUID="608feda4-9f34-43d8-b6bb-e3cc096f7d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 13:20:21.762054 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:21.762012 4931 patch_prober.go:29] interesting pod/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 13:20:21.762251 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:21.762079 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" podUID="608feda4-9f34-43d8-b6bb-e3cc096f7d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 13:20:22.767275 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:22.767214 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" Oct 30 13:20:22.767794 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:22.767771 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" Oct 30 13:20:22.768520 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:22.768499 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" Oct 30 13:20:25.167439 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.167401 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k"] Oct 30 13:20:25.189188 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.189157 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:20:25.213186 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.213147 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k"] Oct 30 13:20:25.213520 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.213496 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f3338e-openshift-default-dockercfg-7bdq7\"" Oct 30 13:20:25.263624 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.263582 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f314d798-3b21-4d5d-9f42-e1eee98c647c-istio-envoy\") pod \"e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:20:25.263806 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.263661 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg7vd\" (UniqueName: \"kubernetes.io/projected/f314d798-3b21-4d5d-9f42-e1eee98c647c-kube-api-access-rg7vd\") pod \"e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:20:25.263806 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.263701 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f314d798-3b21-4d5d-9f42-e1eee98c647c-workload-socket\") pod \"e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:20:25.265571 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.265535 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f314d798-3b21-4d5d-9f42-e1eee98c647c-workload-certs\") pod \"e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:20:25.265704 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.265602 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f314d798-3b21-4d5d-9f42-e1eee98c647c-istio-podinfo\") pod \"e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:20:25.265752 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.265718 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f314d798-3b21-4d5d-9f42-e1eee98c647c-istio-token\") pod \"e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:20:25.265803 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.265771 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f314d798-3b21-4d5d-9f42-e1eee98c647c-credential-socket\") pod \"e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:20:25.265922 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.265903 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f314d798-3b21-4d5d-9f42-e1eee98c647c-istiod-ca-cert\") pod \"e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:20:25.265999 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.265983 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f314d798-3b21-4d5d-9f42-e1eee98c647c-istio-data\") pod \"e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:20:25.368314 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.367322 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f314d798-3b21-4d5d-9f42-e1eee98c647c-istio-envoy\") pod \"e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:20:25.368314 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.367395 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg7vd\" (UniqueName: \"kubernetes.io/projected/f314d798-3b21-4d5d-9f42-e1eee98c647c-kube-api-access-rg7vd\") pod \"e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:20:25.368314 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.367429 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f314d798-3b21-4d5d-9f42-e1eee98c647c-workload-socket\") pod \"e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:20:25.368314 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.367479 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f314d798-3b21-4d5d-9f42-e1eee98c647c-workload-certs\") pod \"e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:20:25.368314 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.367516 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f314d798-3b21-4d5d-9f42-e1eee98c647c-istio-podinfo\") pod \"e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:20:25.368314 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.367569 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f314d798-3b21-4d5d-9f42-e1eee98c647c-istio-token\") pod \"e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:20:25.368314 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.367605 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f314d798-3b21-4d5d-9f42-e1eee98c647c-credential-socket\") pod \"e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:20:25.368314 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.367659 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f314d798-3b21-4d5d-9f42-e1eee98c647c-istiod-ca-cert\") pod \"e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:20:25.368314 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.367710 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f314d798-3b21-4d5d-9f42-e1eee98c647c-istio-data\") pod \"e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:20:25.386348 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.384756 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f314d798-3b21-4d5d-9f42-e1eee98c647c-istio-podinfo\") pod \"e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:20:25.386348 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.385381 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f314d798-3b21-4d5d-9f42-e1eee98c647c-workload-socket\") pod \"e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:20:25.386348 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.385634 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f314d798-3b21-4d5d-9f42-e1eee98c647c-workload-certs\") pod \"e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:20:25.394316 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.392536 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f314d798-3b21-4d5d-9f42-e1eee98c647c-credential-socket\") pod \"e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:20:25.394316 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.393462 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f314d798-3b21-4d5d-9f42-e1eee98c647c-istiod-ca-cert\") pod \"e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:20:25.394316 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.393718 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f314d798-3b21-4d5d-9f42-e1eee98c647c-istio-data\") pod \"e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:20:25.426740 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.419157 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f314d798-3b21-4d5d-9f42-e1eee98c647c-istio-envoy\") pod \"e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:20:25.431064 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.431031 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f314d798-3b21-4d5d-9f42-e1eee98c647c-istio-token\") pod \"e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:20:25.442048 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.439092 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg7vd\" (UniqueName: \"kubernetes.io/projected/f314d798-3b21-4d5d-9f42-e1eee98c647c-kube-api-access-rg7vd\") pod \"e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:20:25.519825 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.519790 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:20:25.868014 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.867984 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k"] Oct 30 13:20:25.869009 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:20:25.868980 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf314d798_3b21_4d5d_9f42_e1eee98c647c.slice/crio-7a7dc8025f22f9d6f24850d8326319b194950ccdaae5d5c4b799dfef98911660 WatchSource:0}: Error finding container 7a7dc8025f22f9d6f24850d8326319b194950ccdaae5d5c4b799dfef98911660: Status 404 returned error can't find the container with id 7a7dc8025f22f9d6f24850d8326319b194950ccdaae5d5c4b799dfef98911660 Oct 30 13:20:25.871979 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.871946 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:20:25.872049 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.872017 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:20:25.872049 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:25.872045 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:20:26.719349 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:26.719300 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" event={"ID":"f314d798-3b21-4d5d-9f42-e1eee98c647c","Type":"ContainerStarted","Data":"d161058b9dae5dcd307bb3376e0ce6a632b66d004dd38f2e575814367a69418e"} Oct 30 13:20:26.719872 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:26.719369 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" event={"ID":"f314d798-3b21-4d5d-9f42-e1eee98c647c","Type":"ContainerStarted","Data":"7a7dc8025f22f9d6f24850d8326319b194950ccdaae5d5c4b799dfef98911660"} Oct 30 13:20:26.794486 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:26.794429 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" podStartSLOduration=1.794413311 podStartE2EDuration="1.794413311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:26.780730683 +0000 UTC m=+2125.384229143" watchObservedRunningTime="2026-10-30 13:20:26.794413311 +0000 UTC m=+2125.397911760" Oct 30 13:20:27.520185 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:27.520155 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:20:27.526315 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:27.526256 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:20:27.726823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:27.726790 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:20:27.727765 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:27.727742 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:20:28.203648 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:28.203610 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz" podUID="5c77e909-e679-4e88-8569-b406b4582f75" containerName="main" containerID="cri-o://0ead8176769b627ba26636c9327e0960be0f784ad424b76cb4776d9e707fe2cd" gracePeriod=45 Oct 30 13:20:28.494888 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:28.494860 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz" Oct 30 13:20:28.614381 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:28.614343 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c77e909-e679-4e88-8569-b406b4582f75-home\" (UniqueName: \"kubernetes.io/empty-dir/5c77e909-e679-4e88-8569-b406b4582f75-home\") pod \"5c77e909-e679-4e88-8569-b406b4582f75\" (UID: \"5c77e909-e679-4e88-8569-b406b4582f75\") " Oct 30 13:20:28.614573 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:28.614481 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c77e909-e679-4e88-8569-b406b4582f75-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c77e909-e679-4e88-8569-b406b4582f75-dshm\") pod \"5c77e909-e679-4e88-8569-b406b4582f75\" (UID: \"5c77e909-e679-4e88-8569-b406b4582f75\") " Oct 30 13:20:28.614573 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:28.614544 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c77e909-e679-4e88-8569-b406b4582f75-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c77e909-e679-4e88-8569-b406b4582f75-tls-certs\") pod \"5c77e909-e679-4e88-8569-b406b4582f75\" (UID: \"5c77e909-e679-4e88-8569-b406b4582f75\") " Oct 30 13:20:28.614685 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:28.614589 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c77e909-e679-4e88-8569-b406b4582f75-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c77e909-e679-4e88-8569-b406b4582f75-model-cache\") pod \"5c77e909-e679-4e88-8569-b406b4582f75\" (UID: \"5c77e909-e679-4e88-8569-b406b4582f75\") " Oct 30 13:20:28.614894 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:28.614699 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c77e909-e679-4e88-8569-b406b4582f75-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c77e909-e679-4e88-8569-b406b4582f75-tmp-dir\") pod \"5c77e909-e679-4e88-8569-b406b4582f75\" (UID: \"5c77e909-e679-4e88-8569-b406b4582f75\") " Oct 30 13:20:28.614894 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:28.614748 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c77e909-e679-4e88-8569-b406b4582f75-kube-api-access-s8nn6\" (UniqueName: \"kubernetes.io/projected/5c77e909-e679-4e88-8569-b406b4582f75-kube-api-access-s8nn6\") pod \"5c77e909-e679-4e88-8569-b406b4582f75\" (UID: \"5c77e909-e679-4e88-8569-b406b4582f75\") " Oct 30 13:20:28.616022 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:28.615997 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c77e909-e679-4e88-8569-b406b4582f75-home" pod "5c77e909-e679-4e88-8569-b406b4582f75" (UID: "5c77e909-e679-4e88-8569-b406b4582f75"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:28.616617 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:28.616584 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c77e909-e679-4e88-8569-b406b4582f75-tmp-dir" pod "5c77e909-e679-4e88-8569-b406b4582f75" (UID: "5c77e909-e679-4e88-8569-b406b4582f75"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:28.616617 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:28.616599 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c77e909-e679-4e88-8569-b406b4582f75-model-cache" pod "5c77e909-e679-4e88-8569-b406b4582f75" (UID: "5c77e909-e679-4e88-8569-b406b4582f75"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:28.618315 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:28.618259 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c77e909-e679-4e88-8569-b406b4582f75-dshm" pod "5c77e909-e679-4e88-8569-b406b4582f75" (UID: "5c77e909-e679-4e88-8569-b406b4582f75"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:28.618763 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:28.618741 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c77e909-e679-4e88-8569-b406b4582f75-tls-certs" pod "5c77e909-e679-4e88-8569-b406b4582f75" (UID: "5c77e909-e679-4e88-8569-b406b4582f75"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:20:28.618930 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:28.618900 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c77e909-e679-4e88-8569-b406b4582f75-kube-api-access-s8nn6" pod "5c77e909-e679-4e88-8569-b406b4582f75" (UID: "5c77e909-e679-4e88-8569-b406b4582f75"). InnerVolumeSpecName "kube-api-access-s8nn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:20:28.717726 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:28.717632 4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c77e909-e679-4e88-8569-b406b4582f75-home\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:20:28.717726 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:28.717669 4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c77e909-e679-4e88-8569-b406b4582f75-dshm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:20:28.717726 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:28.717679 4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c77e909-e679-4e88-8569-b406b4582f75-tls-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:20:28.717726 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:28.717688 4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c77e909-e679-4e88-8569-b406b4582f75-model-cache\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:20:28.717726 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:28.717702 4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c77e909-e679-4e88-8569-b406b4582f75-tmp-dir\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:20:28.717726 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:28.717715 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8nn6\" (UniqueName: \"kubernetes.io/projected/5c77e909-e679-4e88-8569-b406b4582f75-kube-api-access-s8nn6\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:20:28.737513 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:28.737482 4931 generic.go:356] "Generic (PLEG): container finished" podID="5c77e909-e679-4e88-8569-b406b4582f75" containerID="0ead8176769b627ba26636c9327e0960be0f784ad424b76cb4776d9e707fe2cd" exitCode=0 Oct 30 13:20:28.737921 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:28.737528 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz" Oct 30 13:20:28.737921 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:28.737569 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz" event={"ID":"5c77e909-e679-4e88-8569-b406b4582f75","Type":"ContainerDied","Data":"0ead8176769b627ba26636c9327e0960be0f784ad424b76cb4776d9e707fe2cd"} Oct 30 13:20:28.737921 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:28.737622 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz" event={"ID":"5c77e909-e679-4e88-8569-b406b4582f75","Type":"ContainerDied","Data":"39177a3c3a0987ce2deb015c94744c885bd6b6fc01778c4ab2b498a485166268"} Oct 30 13:20:28.737921 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:28.737645 4931 scope.go:122] "RemoveContainer" containerID="0ead8176769b627ba26636c9327e0960be0f784ad424b76cb4776d9e707fe2cd" Oct 30 13:20:28.750280 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:28.750248 4931 scope.go:122] "RemoveContainer" containerID="0ead8176769b627ba26636c9327e0960be0f784ad424b76cb4776d9e707fe2cd" Oct 30 13:20:28.750637 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:20:28.750614 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ead8176769b627ba26636c9327e0960be0f784ad424b76cb4776d9e707fe2cd\": container with ID starting with 0ead8176769b627ba26636c9327e0960be0f784ad424b76cb4776d9e707fe2cd not found: ID does not exist" containerID="0ead8176769b627ba26636c9327e0960be0f784ad424b76cb4776d9e707fe2cd" Oct 30 13:20:28.750698 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:28.750653 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ead8176769b627ba26636c9327e0960be0f784ad424b76cb4776d9e707fe2cd"} err="failed to get container status \"0ead8176769b627ba26636c9327e0960be0f784ad424b76cb4776d9e707fe2cd\": rpc error: code = NotFound desc = could not find container \"0ead8176769b627ba26636c9327e0960be0f784ad424b76cb4776d9e707fe2cd\": container with ID starting with 0ead8176769b627ba26636c9327e0960be0f784ad424b76cb4776d9e707fe2cd not found: ID does not exist" Oct 30 13:20:28.822793 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:28.822757 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz"] Oct 30 13:20:28.826498 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:28.826469 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-4a8ee1/test-model-4a8ee1-kserve-766c76b76c-wldrz"] Oct 30 13:20:29.515159 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:29.515128 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c77e909-e679-4e88-8569-b406b4582f75" path="/var/lib/kubelet/pods/5c77e909-e679-4e88-8569-b406b4582f75/volumes" Oct 30 13:20:30.727408 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:30.727370 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2"] Oct 30 13:20:37.355170 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:20:37.355131 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:20:44.166057 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:44.166014 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96"] Oct 30 13:20:44.166514 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:44.166320 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" podUID="01524a5a-15e5-45ad-a9cb-15b6d605bf86" containerName="istio-proxy" containerID="cri-o://cb52ab5cbeb1fe1eeb3aeaf4327093fe8c5f08fb1918fdfb545ce9aaac9c939a" gracePeriod=30 Oct 30 13:20:45.749283 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:45.749227 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2" podUID="3eb1597f-ffc6-46f2-a6fb-34e000c3ef83" containerName="main" containerID="cri-o://d4d850c54d777523ace5cc98af21d3645588353a094b1cf2b73e765a8116f907" gracePeriod=45 Oct 30 13:20:45.927344 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:45.927301 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2" event={"ID":"3eb1597f-ffc6-46f2-a6fb-34e000c3ef83","Type":"ContainerDied","Data":"d4d850c54d777523ace5cc98af21d3645588353a094b1cf2b73e765a8116f907"} Oct 30 13:20:45.927523 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:45.927377 4931 generic.go:356] "Generic (PLEG): container finished" podID="3eb1597f-ffc6-46f2-a6fb-34e000c3ef83" containerID="d4d850c54d777523ace5cc98af21d3645588353a094b1cf2b73e765a8116f907" exitCode=0 Oct 30 13:20:46.055855 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:46.055822 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2" Oct 30 13:20:46.230499 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:46.230465 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-dshm\") pod \"3eb1597f-ffc6-46f2-a6fb-34e000c3ef83\" (UID: \"3eb1597f-ffc6-46f2-a6fb-34e000c3ef83\") " Oct 30 13:20:46.230885 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:46.230863 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-kube-api-access-fctwf\" (UniqueName: \"kubernetes.io/projected/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-kube-api-access-fctwf\") pod \"3eb1597f-ffc6-46f2-a6fb-34e000c3ef83\" (UID: \"3eb1597f-ffc6-46f2-a6fb-34e000c3ef83\") " Oct 30 13:20:46.231227 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:46.231211 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-tls-certs\") pod \"3eb1597f-ffc6-46f2-a6fb-34e000c3ef83\" (UID: \"3eb1597f-ffc6-46f2-a6fb-34e000c3ef83\") " Oct 30 13:20:46.231656 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:46.231637 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-model-cache\") pod \"3eb1597f-ffc6-46f2-a6fb-34e000c3ef83\" (UID: \"3eb1597f-ffc6-46f2-a6fb-34e000c3ef83\") " Oct 30 13:20:46.231878 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:46.231857 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-home\" (UniqueName: \"kubernetes.io/empty-dir/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-home\") pod \"3eb1597f-ffc6-46f2-a6fb-34e000c3ef83\" (UID: \"3eb1597f-ffc6-46f2-a6fb-34e000c3ef83\") " Oct 30 13:20:46.232044 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:46.232021 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-tmp-dir\") pod \"3eb1597f-ffc6-46f2-a6fb-34e000c3ef83\" (UID: \"3eb1597f-ffc6-46f2-a6fb-34e000c3ef83\") " Oct 30 13:20:46.233375 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:46.233339 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-tmp-dir" pod "3eb1597f-ffc6-46f2-a6fb-34e000c3ef83" (UID: "3eb1597f-ffc6-46f2-a6fb-34e000c3ef83"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:46.233827 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:46.233806 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-model-cache" pod "3eb1597f-ffc6-46f2-a6fb-34e000c3ef83" (UID: "3eb1597f-ffc6-46f2-a6fb-34e000c3ef83"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:46.241130 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:46.240607 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-home" pod "3eb1597f-ffc6-46f2-a6fb-34e000c3ef83" (UID: "3eb1597f-ffc6-46f2-a6fb-34e000c3ef83"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:46.247588 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:46.247551 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-kube-api-access-fctwf" pod "3eb1597f-ffc6-46f2-a6fb-34e000c3ef83" (UID: "3eb1597f-ffc6-46f2-a6fb-34e000c3ef83"). InnerVolumeSpecName "kube-api-access-fctwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:20:46.248073 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:46.248051 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-dshm" pod "3eb1597f-ffc6-46f2-a6fb-34e000c3ef83" (UID: "3eb1597f-ffc6-46f2-a6fb-34e000c3ef83"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:46.248192 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:46.248110 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-tls-certs" pod "3eb1597f-ffc6-46f2-a6fb-34e000c3ef83" (UID: "3eb1597f-ffc6-46f2-a6fb-34e000c3ef83"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:20:46.337998 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:46.337935 4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-tls-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:20:46.338317 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:46.338299 4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-model-cache\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:20:46.338473 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:46.338460 4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-home\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:20:46.338602 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:46.338575 4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-tmp-dir\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:20:46.338704 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:46.338694 4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-dshm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:20:46.338828 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:46.338816 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fctwf\" (UniqueName: \"kubernetes.io/projected/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83-kube-api-access-fctwf\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:20:46.936444 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:46.936408 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2" Oct 30 13:20:46.936961 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:46.936436 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2" event={"ID":"3eb1597f-ffc6-46f2-a6fb-34e000c3ef83","Type":"ContainerDied","Data":"d3e9a705446ee1ee9ead89f72ace845cc0796572843bfe5be973a79a98e9ed94"} Oct 30 13:20:46.936961 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:46.936507 4931 scope.go:122] "RemoveContainer" containerID="d4d850c54d777523ace5cc98af21d3645588353a094b1cf2b73e765a8116f907" Oct 30 13:20:47.045301 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:47.042738 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2"] Oct 30 13:20:47.057295 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:47.055105 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-6f9071/test-model-6f9071-kserve-849958c557-x72j2"] Oct 30 13:20:47.665313 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:47.658984 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eb1597f-ffc6-46f2-a6fb-34e000c3ef83" path="/var/lib/kubelet/pods/3eb1597f-ffc6-46f2-a6fb-34e000c3ef83/volumes" Oct 30 13:20:48.212738 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:48.212700 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-q8w96 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:20:48.213127 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:48.212774 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" podUID="01524a5a-15e5-45ad-a9cb-15b6d605bf86" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:20:49.505166 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:49.505134 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" Oct 30 13:20:49.550103 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:49.542736 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01524a5a-15e5-45ad-a9cb-15b6d605bf86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01524a5a-15e5-45ad-a9cb-15b6d605bf86-istio-data\") pod \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " Oct 30 13:20:49.550103 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:49.542813 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01524a5a-15e5-45ad-a9cb-15b6d605bf86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01524a5a-15e5-45ad-a9cb-15b6d605bf86-istio-envoy\") pod \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " Oct 30 13:20:49.550103 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:49.542892 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01524a5a-15e5-45ad-a9cb-15b6d605bf86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01524a5a-15e5-45ad-a9cb-15b6d605bf86-istio-podinfo\") pod \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " Oct 30 13:20:49.550103 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:49.543020 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01524a5a-15e5-45ad-a9cb-15b6d605bf86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01524a5a-15e5-45ad-a9cb-15b6d605bf86-workload-certs\") pod \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " Oct 30 13:20:49.550103 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:49.543124 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01524a5a-15e5-45ad-a9cb-15b6d605bf86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01524a5a-15e5-45ad-a9cb-15b6d605bf86-workload-socket\") pod \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " Oct 30 13:20:49.550103 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:49.543176 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01524a5a-15e5-45ad-a9cb-15b6d605bf86-istio-token\" (UniqueName: \"kubernetes.io/projected/01524a5a-15e5-45ad-a9cb-15b6d605bf86-istio-token\") pod \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " Oct 30 13:20:49.550103 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:49.543234 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01524a5a-15e5-45ad-a9cb-15b6d605bf86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01524a5a-15e5-45ad-a9cb-15b6d605bf86-credential-socket\") pod \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " Oct 30 13:20:49.550103 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:49.543351 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01524a5a-15e5-45ad-a9cb-15b6d605bf86-kube-api-access-5sgb2\" (UniqueName: \"kubernetes.io/projected/01524a5a-15e5-45ad-a9cb-15b6d605bf86-kube-api-access-5sgb2\") pod \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " Oct 30 13:20:49.550103 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:49.543435 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01524a5a-15e5-45ad-a9cb-15b6d605bf86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01524a5a-15e5-45ad-a9cb-15b6d605bf86-istiod-ca-cert\") pod \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\" (UID: \"01524a5a-15e5-45ad-a9cb-15b6d605bf86\") " Oct 30 13:20:49.550103 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:49.544495 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01524a5a-15e5-45ad-a9cb-15b6d605bf86-istiod-ca-cert" pod "01524a5a-15e5-45ad-a9cb-15b6d605bf86" (UID: "01524a5a-15e5-45ad-a9cb-15b6d605bf86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:20:49.550103 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:49.544797 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01524a5a-15e5-45ad-a9cb-15b6d605bf86-istio-data" pod "01524a5a-15e5-45ad-a9cb-15b6d605bf86" (UID: "01524a5a-15e5-45ad-a9cb-15b6d605bf86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:49.550103 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:49.546469 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01524a5a-15e5-45ad-a9cb-15b6d605bf86-workload-socket" pod "01524a5a-15e5-45ad-a9cb-15b6d605bf86" (UID: "01524a5a-15e5-45ad-a9cb-15b6d605bf86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:49.550103 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:49.547011 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01524a5a-15e5-45ad-a9cb-15b6d605bf86-credential-socket" pod "01524a5a-15e5-45ad-a9cb-15b6d605bf86" (UID: "01524a5a-15e5-45ad-a9cb-15b6d605bf86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:49.550103 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:49.547502 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01524a5a-15e5-45ad-a9cb-15b6d605bf86-workload-certs" pod "01524a5a-15e5-45ad-a9cb-15b6d605bf86" (UID: "01524a5a-15e5-45ad-a9cb-15b6d605bf86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:49.608207 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:49.606001 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01524a5a-15e5-45ad-a9cb-15b6d605bf86-kube-api-access-5sgb2" pod "01524a5a-15e5-45ad-a9cb-15b6d605bf86" (UID: "01524a5a-15e5-45ad-a9cb-15b6d605bf86"). InnerVolumeSpecName "kube-api-access-5sgb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:20:49.608207 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:49.607180 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01524a5a-15e5-45ad-a9cb-15b6d605bf86-istio-token" pod "01524a5a-15e5-45ad-a9cb-15b6d605bf86" (UID: "01524a5a-15e5-45ad-a9cb-15b6d605bf86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:20:49.616733 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:49.616357 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01524a5a-15e5-45ad-a9cb-15b6d605bf86-istio-podinfo" pod "01524a5a-15e5-45ad-a9cb-15b6d605bf86" (UID: "01524a5a-15e5-45ad-a9cb-15b6d605bf86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:20:49.620493 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:49.620454 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01524a5a-15e5-45ad-a9cb-15b6d605bf86-istio-envoy" pod "01524a5a-15e5-45ad-a9cb-15b6d605bf86" (UID: "01524a5a-15e5-45ad-a9cb-15b6d605bf86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:49.666653 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:49.666608 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01524a5a-15e5-45ad-a9cb-15b6d605bf86-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:20:49.682294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:49.678454 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01524a5a-15e5-45ad-a9cb-15b6d605bf86-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:20:49.682294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:49.678492 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01524a5a-15e5-45ad-a9cb-15b6d605bf86-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:20:49.682294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:49.678509 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01524a5a-15e5-45ad-a9cb-15b6d605bf86-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:20:49.682294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:49.678523 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01524a5a-15e5-45ad-a9cb-15b6d605bf86-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:20:49.682294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:49.678538 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sgb2\" (UniqueName: \"kubernetes.io/projected/01524a5a-15e5-45ad-a9cb-15b6d605bf86-kube-api-access-5sgb2\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:20:49.682294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:49.678555 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01524a5a-15e5-45ad-a9cb-15b6d605bf86-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:20:49.682294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:49.678571 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01524a5a-15e5-45ad-a9cb-15b6d605bf86-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:20:49.682294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:49.678585 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01524a5a-15e5-45ad-a9cb-15b6d605bf86-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:20:49.979888 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:49.979855 4931 generic.go:356] "Generic (PLEG): container finished" podID="01524a5a-15e5-45ad-a9cb-15b6d605bf86" containerID="cb52ab5cbeb1fe1eeb3aeaf4327093fe8c5f08fb1918fdfb545ce9aaac9c939a" exitCode=0 Oct 30 13:20:49.980177 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:49.980158 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" Oct 30 13:20:49.980315 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:49.979993 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" event={"ID":"01524a5a-15e5-45ad-a9cb-15b6d605bf86","Type":"ContainerDied","Data":"cb52ab5cbeb1fe1eeb3aeaf4327093fe8c5f08fb1918fdfb545ce9aaac9c939a"} Oct 30 13:20:49.980451 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:49.980435 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96" event={"ID":"01524a5a-15e5-45ad-a9cb-15b6d605bf86","Type":"ContainerDied","Data":"bca7b90f276f189ad4c84f8c3fc9cd9a1dce282b6d325c10bc7dac6fa3a9198a"} Oct 30 13:20:49.980548 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:49.980520 4931 scope.go:122] "RemoveContainer" containerID="cb52ab5cbeb1fe1eeb3aeaf4327093fe8c5f08fb1918fdfb545ce9aaac9c939a" Oct 30 13:20:49.993443 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:49.993422 4931 scope.go:122] "RemoveContainer" containerID="cb52ab5cbeb1fe1eeb3aeaf4327093fe8c5f08fb1918fdfb545ce9aaac9c939a" Oct 30 13:20:49.994026 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:20:49.994001 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb52ab5cbeb1fe1eeb3aeaf4327093fe8c5f08fb1918fdfb545ce9aaac9c939a\": container with ID starting with cb52ab5cbeb1fe1eeb3aeaf4327093fe8c5f08fb1918fdfb545ce9aaac9c939a not found: ID does not exist" containerID="cb52ab5cbeb1fe1eeb3aeaf4327093fe8c5f08fb1918fdfb545ce9aaac9c939a" Oct 30 13:20:49.994156 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:49.994135 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb52ab5cbeb1fe1eeb3aeaf4327093fe8c5f08fb1918fdfb545ce9aaac9c939a"} err="failed to get container status \"cb52ab5cbeb1fe1eeb3aeaf4327093fe8c5f08fb1918fdfb545ce9aaac9c939a\": rpc error: code = NotFound desc = could not find container \"cb52ab5cbeb1fe1eeb3aeaf4327093fe8c5f08fb1918fdfb545ce9aaac9c939a\": container with ID starting with cb52ab5cbeb1fe1eeb3aeaf4327093fe8c5f08fb1918fdfb545ce9aaac9c939a not found: ID does not exist" Oct 30 13:20:50.069401 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:50.069357 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96"] Oct 30 13:20:50.073138 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:50.073104 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96"] Oct 30 13:20:51.514317 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:51.514287 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01524a5a-15e5-45ad-a9cb-15b6d605bf86" path="/var/lib/kubelet/pods/01524a5a-15e5-45ad-a9cb-15b6d605bf86/volumes" Oct 30 13:20:56.501540 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:56.501499 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np"] Oct 30 13:20:56.502047 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:56.501757 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np" podUID="7c31718e-7be5-4591-b32b-bf9655c06c2a" containerName="maas-api" containerID="cri-o://64a185a6f3369676255fcdca7e8bcb76ea453690e0a092f83616a43727af1fb8" gracePeriod=30 Oct 30 13:20:57.004497 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:57.004459 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np" Oct 30 13:20:57.099461 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:57.099421 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c31718e-7be5-4591-b32b-bf9655c06c2a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c31718e-7be5-4591-b32b-bf9655c06c2a-openshift-service-ca\") pod \"7c31718e-7be5-4591-b32b-bf9655c06c2a\" (UID: \"7c31718e-7be5-4591-b32b-bf9655c06c2a\") " Oct 30 13:20:57.099723 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:57.099503 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c31718e-7be5-4591-b32b-bf9655c06c2a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c31718e-7be5-4591-b32b-bf9655c06c2a-metrics-certs\") pod \"7c31718e-7be5-4591-b32b-bf9655c06c2a\" (UID: \"7c31718e-7be5-4591-b32b-bf9655c06c2a\") " Oct 30 13:20:57.099723 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:57.099611 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c31718e-7be5-4591-b32b-bf9655c06c2a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c31718e-7be5-4591-b32b-bf9655c06c2a-maas-api-tls\") pod \"7c31718e-7be5-4591-b32b-bf9655c06c2a\" (UID: \"7c31718e-7be5-4591-b32b-bf9655c06c2a\") " Oct 30 13:20:57.099723 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:57.099651 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c31718e-7be5-4591-b32b-bf9655c06c2a-kube-api-access-94vj8\" (UniqueName: \"kubernetes.io/projected/7c31718e-7be5-4591-b32b-bf9655c06c2a-kube-api-access-94vj8\") pod \"7c31718e-7be5-4591-b32b-bf9655c06c2a\" (UID: \"7c31718e-7be5-4591-b32b-bf9655c06c2a\") " Oct 30 13:20:57.106469 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:57.106407 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c31718e-7be5-4591-b32b-bf9655c06c2a-openshift-service-ca" pod "7c31718e-7be5-4591-b32b-bf9655c06c2a" (UID: "7c31718e-7be5-4591-b32b-bf9655c06c2a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:20:57.124434 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:57.123155 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c31718e-7be5-4591-b32b-bf9655c06c2a-kube-api-access-94vj8" pod "7c31718e-7be5-4591-b32b-bf9655c06c2a" (UID: "7c31718e-7be5-4591-b32b-bf9655c06c2a"). InnerVolumeSpecName "kube-api-access-94vj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:20:57.124434 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:57.123531 4931 generic.go:356] "Generic (PLEG): container finished" podID="7c31718e-7be5-4591-b32b-bf9655c06c2a" containerID="64a185a6f3369676255fcdca7e8bcb76ea453690e0a092f83616a43727af1fb8" exitCode=0 Oct 30 13:20:57.124434 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:57.123782 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np" event={"ID":"7c31718e-7be5-4591-b32b-bf9655c06c2a","Type":"ContainerDied","Data":"64a185a6f3369676255fcdca7e8bcb76ea453690e0a092f83616a43727af1fb8"} Oct 30 13:20:57.124434 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:57.123881 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np" event={"ID":"7c31718e-7be5-4591-b32b-bf9655c06c2a","Type":"ContainerDied","Data":"84d2952ea9b3bd12b1ff09032f61271fbc3a679a5349c58c70cbc47d92bd5202"} Oct 30 13:20:57.124434 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:57.123919 4931 scope.go:122] "RemoveContainer" containerID="64a185a6f3369676255fcdca7e8bcb76ea453690e0a092f83616a43727af1fb8" Oct 30 13:20:57.124434 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:57.124140 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np" Oct 30 13:20:57.126228 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:57.125187 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c31718e-7be5-4591-b32b-bf9655c06c2a-maas-api-tls" pod "7c31718e-7be5-4591-b32b-bf9655c06c2a" (UID: "7c31718e-7be5-4591-b32b-bf9655c06c2a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:20:57.126756 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:57.126725 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c31718e-7be5-4591-b32b-bf9655c06c2a-metrics-certs" pod "7c31718e-7be5-4591-b32b-bf9655c06c2a" (UID: "7c31718e-7be5-4591-b32b-bf9655c06c2a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:20:57.153015 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:57.152984 4931 scope.go:122] "RemoveContainer" containerID="64a185a6f3369676255fcdca7e8bcb76ea453690e0a092f83616a43727af1fb8" Oct 30 13:20:57.153461 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:20:57.153428 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64a185a6f3369676255fcdca7e8bcb76ea453690e0a092f83616a43727af1fb8\": container with ID starting with 64a185a6f3369676255fcdca7e8bcb76ea453690e0a092f83616a43727af1fb8 not found: ID does not exist" containerID="64a185a6f3369676255fcdca7e8bcb76ea453690e0a092f83616a43727af1fb8" Oct 30 13:20:57.153593 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:57.153472 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64a185a6f3369676255fcdca7e8bcb76ea453690e0a092f83616a43727af1fb8"} err="failed to get container status \"64a185a6f3369676255fcdca7e8bcb76ea453690e0a092f83616a43727af1fb8\": rpc error: code = NotFound desc = could not find container \"64a185a6f3369676255fcdca7e8bcb76ea453690e0a092f83616a43727af1fb8\": container with ID starting with 64a185a6f3369676255fcdca7e8bcb76ea453690e0a092f83616a43727af1fb8 not found: ID does not exist" Oct 30 13:20:57.201961 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:57.201918 4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c31718e-7be5-4591-b32b-bf9655c06c2a-metrics-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:20:57.201961 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:57.201961 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c31718e-7be5-4591-b32b-bf9655c06c2a-maas-api-tls\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:20:57.202177 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:57.201977 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94vj8\" (UniqueName: \"kubernetes.io/projected/7c31718e-7be5-4591-b32b-bf9655c06c2a-kube-api-access-94vj8\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:20:57.202177 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:57.201991 4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c31718e-7be5-4591-b32b-bf9655c06c2a-openshift-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:20:57.477608 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:57.477577 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np"] Oct 30 13:20:57.482725 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:57.482700 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4a8ee1-b5db7d855-kl2np"] Oct 30 13:20:57.533136 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:57.533092 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c31718e-7be5-4591-b32b-bf9655c06c2a" path="/var/lib/kubelet/pods/7c31718e-7be5-4591-b32b-bf9655c06c2a/volumes" Oct 30 13:20:58.133217 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:58.133183 4931 generic.go:356] "Generic (PLEG): container finished" podID="212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf" containerID="f37ca073e38d4e6229125b4e00a5e9b5222822eaa461c4de7621af75389f1f98" exitCode=0 Oct 30 13:20:58.133416 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:58.133262 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" event={"ID":"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf","Type":"ContainerDied","Data":"f37ca073e38d4e6229125b4e00a5e9b5222822eaa461c4de7621af75389f1f98"} Oct 30 13:20:58.133836 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:58.133807 4931 scope.go:122] "RemoveContainer" containerID="f37ca073e38d4e6229125b4e00a5e9b5222822eaa461c4de7621af75389f1f98" Oct 30 13:20:58.135789 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:58.135764 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:20:58.135863 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:58.135831 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:20:58.135901 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:58.135861 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:20:59.158291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:59.157405 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" event={"ID":"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf","Type":"ContainerStarted","Data":"8c2c976c62c4e746a750e298b26681d455e45b77d7ff01162f8305916226ee33"} Oct 30 13:20:59.279800 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:59.279762 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:20:59.280977 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:59.280945 4931 patch_prober.go:29] interesting pod/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 13:20:59.281117 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:20:59.281006 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" podUID="212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 13:21:00.280298 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:21:00.280240 4931 patch_prober.go:29] interesting pod/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 13:21:00.280696 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:21:00.280323 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" podUID="212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 13:21:01.280030 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:21:01.279985 4931 patch_prober.go:29] interesting pod/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 13:21:01.280223 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:21:01.280058 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" podUID="212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 13:21:02.279837 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:21:02.279800 4931 patch_prober.go:29] interesting pod/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 13:21:02.280253 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:21:02.279863 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" podUID="212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 13:21:02.817552 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:21:02.817521 4931 scope.go:122] "RemoveContainer" containerID="4539c9c0d36d2a7a5b72d2d661becd7739d2c7cba9c8f5e3ecdb0cff6c995273" Oct 30 13:21:02.828429 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:21:02.828404 4931 scope.go:122] "RemoveContainer" containerID="c1f39d7956890aefc615831680faeaf153791c9868a87e28e3e3e00585b304b9" Oct 30 13:21:02.883103 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:21:02.883078 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:21:03.279826 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:21:03.279787 4931 patch_prober.go:29] interesting pod/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 13:21:03.280023 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:21:03.279843 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" podUID="212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 13:21:04.280018 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:21:04.279979 4931 patch_prober.go:29] interesting pod/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 13:21:04.280568 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:21:04.280042 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" podUID="212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 13:21:05.285401 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:21:05.285362 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:21:05.287044 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:21:05.287015 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:21:07.358428 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:21:07.358381 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:21:31.373640 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:21:31.373602 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:21:37.352153 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:21:37.352115 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:22:07.354062 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:22:07.354023 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:22:25.973430 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:25.973391 4931 generic.go:356] "Generic (PLEG): container finished" podID="f314d798-3b21-4d5d-9f42-e1eee98c647c" containerID="d161058b9dae5dcd307bb3376e0ce6a632b66d004dd38f2e575814367a69418e" exitCode=0 Oct 30 13:22:25.974074 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:25.973464 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" event={"ID":"f314d798-3b21-4d5d-9f42-e1eee98c647c","Type":"ContainerDied","Data":"d161058b9dae5dcd307bb3376e0ce6a632b66d004dd38f2e575814367a69418e"} Oct 30 13:22:25.974074 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:25.973933 4931 scope.go:122] "RemoveContainer" containerID="d161058b9dae5dcd307bb3376e0ce6a632b66d004dd38f2e575814367a69418e" Oct 30 13:22:25.976209 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:25.976182 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:22:25.976310 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:25.976291 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:22:25.976365 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:25.976347 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:22:26.982051 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:26.982014 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" event={"ID":"f314d798-3b21-4d5d-9f42-e1eee98c647c","Type":"ContainerStarted","Data":"c5ee9c9e0b15271c1ad7397bbd81d05bb3843220349bf8028af678ccd3cf16e1"} Oct 30 13:22:27.541331 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:27.541294 4931 patch_prober.go:29] interesting pod/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 13:22:27.544053 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:27.544012 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" podUID="f314d798-3b21-4d5d-9f42-e1eee98c647c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 13:22:27.559290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:27.552077 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:22:27.727376 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:27.727337 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:22:28.520582 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:28.520544 4931 patch_prober.go:29] interesting pod/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 13:22:28.520968 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:28.520608 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" podUID="f314d798-3b21-4d5d-9f42-e1eee98c647c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 13:22:29.520858 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:29.520827 4931 patch_prober.go:29] interesting pod/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 13:22:29.521333 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:29.520877 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" podUID="f314d798-3b21-4d5d-9f42-e1eee98c647c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 13:22:30.520559 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:30.520524 4931 patch_prober.go:29] interesting pod/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 13:22:30.520743 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:30.520576 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" podUID="f314d798-3b21-4d5d-9f42-e1eee98c647c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 13:22:31.520362 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:31.520322 4931 patch_prober.go:29] interesting pod/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 13:22:31.520787 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:31.520378 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" podUID="f314d798-3b21-4d5d-9f42-e1eee98c647c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 13:22:32.527325 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:32.527249 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:22:32.528371 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:32.528349 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:22:33.265261 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:33.265226 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd"] Oct 30 13:22:37.354639 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:22:37.354599 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:22:48.286685 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:48.286645 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd" podUID="2023086a-0026-41a0-9180-d48b4523e392" containerName="main" containerID="cri-o://ce620dd132925e305485ef5ffdcd064e3e31d7d27def84562213dfff4ec4ea12" gracePeriod=45 Oct 30 13:22:48.576877 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:48.576852 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd" Oct 30 13:22:48.672584 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:48.672546 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2023086a-0026-41a0-9180-d48b4523e392-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2023086a-0026-41a0-9180-d48b4523e392-tmp-dir\") pod \"2023086a-0026-41a0-9180-d48b4523e392\" (UID: \"2023086a-0026-41a0-9180-d48b4523e392\") " Oct 30 13:22:48.672746 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:48.672610 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2023086a-0026-41a0-9180-d48b4523e392-kube-api-access-2xt4v\" (UniqueName: \"kubernetes.io/projected/2023086a-0026-41a0-9180-d48b4523e392-kube-api-access-2xt4v\") pod \"2023086a-0026-41a0-9180-d48b4523e392\" (UID: \"2023086a-0026-41a0-9180-d48b4523e392\") " Oct 30 13:22:48.672746 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:48.672644 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2023086a-0026-41a0-9180-d48b4523e392-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2023086a-0026-41a0-9180-d48b4523e392-model-cache\") pod \"2023086a-0026-41a0-9180-d48b4523e392\" (UID: \"2023086a-0026-41a0-9180-d48b4523e392\") " Oct 30 13:22:48.672746 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:48.672719 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2023086a-0026-41a0-9180-d48b4523e392-home\" (UniqueName: \"kubernetes.io/empty-dir/2023086a-0026-41a0-9180-d48b4523e392-home\") pod \"2023086a-0026-41a0-9180-d48b4523e392\" (UID: \"2023086a-0026-41a0-9180-d48b4523e392\") " Oct 30 13:22:48.672922 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:48.672764 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2023086a-0026-41a0-9180-d48b4523e392-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2023086a-0026-41a0-9180-d48b4523e392-dshm\") pod \"2023086a-0026-41a0-9180-d48b4523e392\" (UID: \"2023086a-0026-41a0-9180-d48b4523e392\") " Oct 30 13:22:48.672922 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:48.672838 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2023086a-0026-41a0-9180-d48b4523e392-tmp-dir" pod "2023086a-0026-41a0-9180-d48b4523e392" (UID: "2023086a-0026-41a0-9180-d48b4523e392"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:22:48.673033 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:48.672953 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2023086a-0026-41a0-9180-d48b4523e392-tls-certs\" (UniqueName: \"kubernetes.io/secret/2023086a-0026-41a0-9180-d48b4523e392-tls-certs\") pod \"2023086a-0026-41a0-9180-d48b4523e392\" (UID: \"2023086a-0026-41a0-9180-d48b4523e392\") " Oct 30 13:22:48.673090 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:48.673032 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2023086a-0026-41a0-9180-d48b4523e392-home" pod "2023086a-0026-41a0-9180-d48b4523e392" (UID: "2023086a-0026-41a0-9180-d48b4523e392"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:22:48.673090 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:48.673045 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2023086a-0026-41a0-9180-d48b4523e392-model-cache" pod "2023086a-0026-41a0-9180-d48b4523e392" (UID: "2023086a-0026-41a0-9180-d48b4523e392"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:22:48.673550 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:48.673525 4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2023086a-0026-41a0-9180-d48b4523e392-tmp-dir\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:22:48.673550 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:48.673546 4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2023086a-0026-41a0-9180-d48b4523e392-model-cache\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:22:48.673550 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:48.673557 4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2023086a-0026-41a0-9180-d48b4523e392-home\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:22:48.675132 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:48.675108 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2023086a-0026-41a0-9180-d48b4523e392-dshm" pod "2023086a-0026-41a0-9180-d48b4523e392" (UID: "2023086a-0026-41a0-9180-d48b4523e392"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:22:48.675313 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:48.675228 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2023086a-0026-41a0-9180-d48b4523e392-kube-api-access-2xt4v" pod "2023086a-0026-41a0-9180-d48b4523e392" (UID: "2023086a-0026-41a0-9180-d48b4523e392"). InnerVolumeSpecName "kube-api-access-2xt4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:22:48.675610 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:48.675587 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2023086a-0026-41a0-9180-d48b4523e392-tls-certs" pod "2023086a-0026-41a0-9180-d48b4523e392" (UID: "2023086a-0026-41a0-9180-d48b4523e392"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:22:48.774958 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:48.774928 4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2023086a-0026-41a0-9180-d48b4523e392-tls-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:22:48.774958 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:48.774955 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xt4v\" (UniqueName: \"kubernetes.io/projected/2023086a-0026-41a0-9180-d48b4523e392-kube-api-access-2xt4v\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:22:48.775139 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:48.774966 4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2023086a-0026-41a0-9180-d48b4523e392-dshm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:22:49.199990 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:49.199959 4931 generic.go:356] "Generic (PLEG): container finished" podID="2023086a-0026-41a0-9180-d48b4523e392" containerID="ce620dd132925e305485ef5ffdcd064e3e31d7d27def84562213dfff4ec4ea12" exitCode=0 Oct 30 13:22:49.200189 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:49.200000 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd" Oct 30 13:22:49.200189 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:49.200048 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd" event={"ID":"2023086a-0026-41a0-9180-d48b4523e392","Type":"ContainerDied","Data":"ce620dd132925e305485ef5ffdcd064e3e31d7d27def84562213dfff4ec4ea12"} Oct 30 13:22:49.200189 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:49.200099 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd" event={"ID":"2023086a-0026-41a0-9180-d48b4523e392","Type":"ContainerDied","Data":"690c0a4b0ad153bc0b1001316da330d993879377eb9d89ccdbdc0075a68071dd"} Oct 30 13:22:49.200189 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:49.200118 4931 scope.go:122] "RemoveContainer" containerID="ce620dd132925e305485ef5ffdcd064e3e31d7d27def84562213dfff4ec4ea12" Oct 30 13:22:49.211254 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:49.211238 4931 scope.go:122] "RemoveContainer" containerID="ce620dd132925e305485ef5ffdcd064e3e31d7d27def84562213dfff4ec4ea12" Oct 30 13:22:49.211553 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:22:49.211533 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce620dd132925e305485ef5ffdcd064e3e31d7d27def84562213dfff4ec4ea12\": container with ID starting with ce620dd132925e305485ef5ffdcd064e3e31d7d27def84562213dfff4ec4ea12 not found: ID does not exist" containerID="ce620dd132925e305485ef5ffdcd064e3e31d7d27def84562213dfff4ec4ea12" Oct 30 13:22:49.211626 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:49.211561 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce620dd132925e305485ef5ffdcd064e3e31d7d27def84562213dfff4ec4ea12"} err="failed to get container status \"ce620dd132925e305485ef5ffdcd064e3e31d7d27def84562213dfff4ec4ea12\": rpc error: code = NotFound desc = could not find container \"ce620dd132925e305485ef5ffdcd064e3e31d7d27def84562213dfff4ec4ea12\": container with ID starting with ce620dd132925e305485ef5ffdcd064e3e31d7d27def84562213dfff4ec4ea12 not found: ID does not exist" Oct 30 13:22:49.258054 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:49.258028 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd"] Oct 30 13:22:49.271441 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:49.271419 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-cd2d2e/test-model-cd2d2e-kserve-7d8c46bfbc-2htxd"] Oct 30 13:22:49.514903 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:49.514818 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2023086a-0026-41a0-9180-d48b4523e392" path="/var/lib/kubelet/pods/2023086a-0026-41a0-9180-d48b4523e392/volumes" Oct 30 13:22:54.565163 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:22:54.565129 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:23:02.525456 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:02.525403 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d"] Oct 30 13:23:02.526674 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:02.525697 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d" podUID="1a9754b7-dc87-483f-9164-037a383d1f99" containerName="maas-api" containerID="cri-o://0b663915cd9de167730ed79227bb1a23ce289e127d088d51042b7ae6df025d7e" gracePeriod=30 Oct 30 13:23:02.987338 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:02.985829 4931 scope.go:122] "RemoveContainer" containerID="58dfec2f3c35b81d7f0b2d69e47880ef34d27afdff8a2fca0aa4a42d39e60e55" Oct 30 13:23:03.028668 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:03.028627 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d" Oct 30 13:23:03.039286 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:03.034431 4931 scope.go:122] "RemoveContainer" containerID="0b663915cd9de167730ed79227bb1a23ce289e127d088d51042b7ae6df025d7e" Oct 30 13:23:03.100625 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:03.100592 4931 scope.go:122] "RemoveContainer" containerID="05b1d0f687144e9951b6e4f7bd79cd5fbe3afd8d57ef4ec810e651a6efb70aa5" Oct 30 13:23:03.122410 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:03.122378 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a9754b7-dc87-483f-9164-037a383d1f99-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a9754b7-dc87-483f-9164-037a383d1f99-metrics-certs\") pod \"1a9754b7-dc87-483f-9164-037a383d1f99\" (UID: \"1a9754b7-dc87-483f-9164-037a383d1f99\") " Oct 30 13:23:03.122578 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:03.122560 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a9754b7-dc87-483f-9164-037a383d1f99-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a9754b7-dc87-483f-9164-037a383d1f99-maas-api-tls\") pod \"1a9754b7-dc87-483f-9164-037a383d1f99\" (UID: \"1a9754b7-dc87-483f-9164-037a383d1f99\") " Oct 30 13:23:03.122646 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:03.122599 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a9754b7-dc87-483f-9164-037a383d1f99-kube-api-access-5rpjl\" (UniqueName: \"kubernetes.io/projected/1a9754b7-dc87-483f-9164-037a383d1f99-kube-api-access-5rpjl\") pod \"1a9754b7-dc87-483f-9164-037a383d1f99\" (UID: \"1a9754b7-dc87-483f-9164-037a383d1f99\") " Oct 30 13:23:03.122708 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:03.122673 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a9754b7-dc87-483f-9164-037a383d1f99-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a9754b7-dc87-483f-9164-037a383d1f99-openshift-service-ca\") pod \"1a9754b7-dc87-483f-9164-037a383d1f99\" (UID: \"1a9754b7-dc87-483f-9164-037a383d1f99\") " Oct 30 13:23:03.123502 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:03.123475 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a9754b7-dc87-483f-9164-037a383d1f99-openshift-service-ca" pod "1a9754b7-dc87-483f-9164-037a383d1f99" (UID: "1a9754b7-dc87-483f-9164-037a383d1f99"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:23:03.129013 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:03.128982 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a9754b7-dc87-483f-9164-037a383d1f99-kube-api-access-5rpjl" pod "1a9754b7-dc87-483f-9164-037a383d1f99" (UID: "1a9754b7-dc87-483f-9164-037a383d1f99"). InnerVolumeSpecName "kube-api-access-5rpjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:23:03.129362 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:03.129334 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a9754b7-dc87-483f-9164-037a383d1f99-metrics-certs" pod "1a9754b7-dc87-483f-9164-037a383d1f99" (UID: "1a9754b7-dc87-483f-9164-037a383d1f99"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:23:03.131343 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:03.131317 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a9754b7-dc87-483f-9164-037a383d1f99-maas-api-tls" pod "1a9754b7-dc87-483f-9164-037a383d1f99" (UID: "1a9754b7-dc87-483f-9164-037a383d1f99"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:23:03.224360 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:03.224326 4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a9754b7-dc87-483f-9164-037a383d1f99-metrics-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:23:03.224360 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:03.224355 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a9754b7-dc87-483f-9164-037a383d1f99-maas-api-tls\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:23:03.224360 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:03.224364 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rpjl\" (UniqueName: \"kubernetes.io/projected/1a9754b7-dc87-483f-9164-037a383d1f99-kube-api-access-5rpjl\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:23:03.224609 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:03.224375 4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a9754b7-dc87-483f-9164-037a383d1f99-openshift-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:23:03.315737 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:03.315694 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d" event={"ID":"1a9754b7-dc87-483f-9164-037a383d1f99","Type":"ContainerDied","Data":"0b663915cd9de167730ed79227bb1a23ce289e127d088d51042b7ae6df025d7e"} Oct 30 13:23:03.315928 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:03.315754 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d" event={"ID":"1a9754b7-dc87-483f-9164-037a383d1f99","Type":"ContainerDied","Data":"bac1be8dd6920e011970aa0e3a39ea8ae2c76b6ea771699e3cb4a9034e2be4aa"} Oct 30 13:23:03.315928 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:03.315768 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d" Oct 30 13:23:03.409491 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:03.409457 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d"] Oct 30 13:23:03.413137 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:03.413104 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6af611-6f5f8fc474-5872d"] Oct 30 13:23:03.519026 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:03.518943 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a9754b7-dc87-483f-9164-037a383d1f99" path="/var/lib/kubelet/pods/1a9754b7-dc87-483f-9164-037a383d1f99/volumes" Oct 30 13:23:07.355854 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:23:07.355816 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:23:11.267891 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:11.265750 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz"] Oct 30 13:23:11.267891 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:11.266014 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" podUID="608feda4-9f34-43d8-b6bb-e3cc096f7d5c" containerName="istio-proxy" containerID="cri-o://8ec474d83771971ce7039d5b54d59419a111363f2e5b4b91942502c49dcfc705" gracePeriod=30 Oct 30 13:23:11.794253 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:11.794213 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb"] Oct 30 13:23:16.357322 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:23:16.357283 4931 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod608feda4_9f34_43d8_b6bb_e3cc096f7d5c.slice/crio-conmon-8ec474d83771971ce7039d5b54d59419a111363f2e5b4b91942502c49dcfc705.scope\": RecentStats: unable to find data in memory cache]" Oct 30 13:23:16.533884 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:16.531827 4931 generic.go:356] "Generic (PLEG): container finished" podID="608feda4-9f34-43d8-b6bb-e3cc096f7d5c" containerID="8ec474d83771971ce7039d5b54d59419a111363f2e5b4b91942502c49dcfc705" exitCode=0 Oct 30 13:23:16.533884 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:16.532161 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" event={"ID":"608feda4-9f34-43d8-b6bb-e3cc096f7d5c","Type":"ContainerDied","Data":"8ec474d83771971ce7039d5b54d59419a111363f2e5b4b91942502c49dcfc705"} Oct 30 13:23:16.649617 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:16.649589 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" Oct 30 13:23:16.753261 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:16.753220 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-istiod-ca-cert\") pod \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " Oct 30 13:23:16.753478 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:16.753317 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-workload-socket\") pod \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " Oct 30 13:23:16.753478 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:16.753348 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-istio-token\" (UniqueName: \"kubernetes.io/projected/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-istio-token\") pod \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " Oct 30 13:23:16.753478 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:16.753385 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-istio-podinfo\") pod \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " Oct 30 13:23:16.753478 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:16.753434 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-kube-api-access-mwv58\" (UniqueName: \"kubernetes.io/projected/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-kube-api-access-mwv58\") pod \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " Oct 30 13:23:16.753714 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:16.753498 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-workload-certs\") pod \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " Oct 30 13:23:16.753714 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:16.753619 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-credential-socket\") pod \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " Oct 30 13:23:16.753714 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:16.753659 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-istiod-ca-cert" pod "608feda4-9f34-43d8-b6bb-e3cc096f7d5c" (UID: "608feda4-9f34-43d8-b6bb-e3cc096f7d5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:23:16.753878 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:16.753720 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-istio-envoy\") pod \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " Oct 30 13:23:16.753878 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:16.753841 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-istio-data\") pod \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\" (UID: \"608feda4-9f34-43d8-b6bb-e3cc096f7d5c\") " Oct 30 13:23:16.754353 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:16.754322 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-workload-certs" pod "608feda4-9f34-43d8-b6bb-e3cc096f7d5c" (UID: "608feda4-9f34-43d8-b6bb-e3cc096f7d5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:23:16.754541 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:16.754491 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:23:16.754625 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:16.754548 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:23:16.754625 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:16.754556 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-istio-data" pod "608feda4-9f34-43d8-b6bb-e3cc096f7d5c" (UID: "608feda4-9f34-43d8-b6bb-e3cc096f7d5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:23:16.754785 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:16.754762 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-credential-socket" pod "608feda4-9f34-43d8-b6bb-e3cc096f7d5c" (UID: "608feda4-9f34-43d8-b6bb-e3cc096f7d5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:23:16.754974 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:16.754950 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-workload-socket" pod "608feda4-9f34-43d8-b6bb-e3cc096f7d5c" (UID: "608feda4-9f34-43d8-b6bb-e3cc096f7d5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:23:16.759603 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:16.759574 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-istio-token" pod "608feda4-9f34-43d8-b6bb-e3cc096f7d5c" (UID: "608feda4-9f34-43d8-b6bb-e3cc096f7d5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:23:16.762654 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:16.762625 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-istio-envoy" pod "608feda4-9f34-43d8-b6bb-e3cc096f7d5c" (UID: "608feda4-9f34-43d8-b6bb-e3cc096f7d5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:23:16.763700 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:16.763667 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-istio-podinfo" pod "608feda4-9f34-43d8-b6bb-e3cc096f7d5c" (UID: "608feda4-9f34-43d8-b6bb-e3cc096f7d5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:23:16.763815 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:16.763769 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-kube-api-access-mwv58" pod "608feda4-9f34-43d8-b6bb-e3cc096f7d5c" (UID: "608feda4-9f34-43d8-b6bb-e3cc096f7d5c"). InnerVolumeSpecName "kube-api-access-mwv58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:23:16.855981 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:16.855893 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:23:16.855981 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:16.855926 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:23:16.855981 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:16.855938 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:23:16.855981 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:16.855949 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:23:16.855981 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:16.855961 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:23:16.855981 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:16.855972 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwv58\" (UniqueName: \"kubernetes.io/projected/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-kube-api-access-mwv58\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:23:16.856298 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:16.855987 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608feda4-9f34-43d8-b6bb-e3cc096f7d5c-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:23:17.542358 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:17.542326 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" Oct 30 13:23:17.542845 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:17.542354 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz" event={"ID":"608feda4-9f34-43d8-b6bb-e3cc096f7d5c","Type":"ContainerDied","Data":"b169389fa9fdae0d2d92c67026ff80d0e96150f40d924de16b90b2591a56a85a"} Oct 30 13:23:17.542845 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:17.542421 4931 scope.go:122] "RemoveContainer" containerID="8ec474d83771971ce7039d5b54d59419a111363f2e5b4b91942502c49dcfc705" Oct 30 13:23:17.614538 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:17.614512 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz"] Oct 30 13:23:17.628628 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:17.628601 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz"] Oct 30 13:23:19.514320 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:19.514282 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="608feda4-9f34-43d8-b6bb-e3cc096f7d5c" path="/var/lib/kubelet/pods/608feda4-9f34-43d8-b6bb-e3cc096f7d5c/volumes" Oct 30 13:23:26.817745 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:26.817687 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb" podUID="2c92125c-50d9-4d98-9978-32c7c94eab42" containerName="main" containerID="cri-o://9ba6972a468ecc349a21d58d857071d17571bf2de533a3297a72f3564b6114ed" gracePeriod=45 Oct 30 13:23:27.110444 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:27.110413 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb" Oct 30 13:23:27.217775 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:27.217737 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c92125c-50d9-4d98-9978-32c7c94eab42-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c92125c-50d9-4d98-9978-32c7c94eab42-tmp-dir\") pod \"2c92125c-50d9-4d98-9978-32c7c94eab42\" (UID: \"2c92125c-50d9-4d98-9978-32c7c94eab42\") " Oct 30 13:23:27.217978 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:27.217854 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c92125c-50d9-4d98-9978-32c7c94eab42-home\" (UniqueName: \"kubernetes.io/empty-dir/2c92125c-50d9-4d98-9978-32c7c94eab42-home\") pod \"2c92125c-50d9-4d98-9978-32c7c94eab42\" (UID: \"2c92125c-50d9-4d98-9978-32c7c94eab42\") " Oct 30 13:23:27.218051 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:27.217987 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c92125c-50d9-4d98-9978-32c7c94eab42-kube-api-access-bdjjw\" (UniqueName: \"kubernetes.io/projected/2c92125c-50d9-4d98-9978-32c7c94eab42-kube-api-access-bdjjw\") pod \"2c92125c-50d9-4d98-9978-32c7c94eab42\" (UID: \"2c92125c-50d9-4d98-9978-32c7c94eab42\") " Oct 30 13:23:27.218051 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:27.218031 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c92125c-50d9-4d98-9978-32c7c94eab42-tmp-dir" pod "2c92125c-50d9-4d98-9978-32c7c94eab42" (UID: "2c92125c-50d9-4d98-9978-32c7c94eab42"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:23:27.218159 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:27.218048 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c92125c-50d9-4d98-9978-32c7c94eab42-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c92125c-50d9-4d98-9978-32c7c94eab42-model-cache\") pod \"2c92125c-50d9-4d98-9978-32c7c94eab42\" (UID: \"2c92125c-50d9-4d98-9978-32c7c94eab42\") " Oct 30 13:23:27.218159 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:27.218113 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c92125c-50d9-4d98-9978-32c7c94eab42-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c92125c-50d9-4d98-9978-32c7c94eab42-tls-certs\") pod \"2c92125c-50d9-4d98-9978-32c7c94eab42\" (UID: \"2c92125c-50d9-4d98-9978-32c7c94eab42\") " Oct 30 13:23:27.218159 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:27.218136 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c92125c-50d9-4d98-9978-32c7c94eab42-home" pod "2c92125c-50d9-4d98-9978-32c7c94eab42" (UID: "2c92125c-50d9-4d98-9978-32c7c94eab42"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:23:27.218322 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:27.218203 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c92125c-50d9-4d98-9978-32c7c94eab42-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c92125c-50d9-4d98-9978-32c7c94eab42-dshm\") pod \"2c92125c-50d9-4d98-9978-32c7c94eab42\" (UID: \"2c92125c-50d9-4d98-9978-32c7c94eab42\") " Oct 30 13:23:27.218390 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:27.218329 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c92125c-50d9-4d98-9978-32c7c94eab42-model-cache" pod "2c92125c-50d9-4d98-9978-32c7c94eab42" (UID: "2c92125c-50d9-4d98-9978-32c7c94eab42"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:23:27.218877 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:27.218852 4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c92125c-50d9-4d98-9978-32c7c94eab42-tmp-dir\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:23:27.219006 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:27.218878 4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c92125c-50d9-4d98-9978-32c7c94eab42-home\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:23:27.219006 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:27.218893 4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c92125c-50d9-4d98-9978-32c7c94eab42-model-cache\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:23:27.220877 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:27.220852 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c92125c-50d9-4d98-9978-32c7c94eab42-tls-certs" pod "2c92125c-50d9-4d98-9978-32c7c94eab42" (UID: "2c92125c-50d9-4d98-9978-32c7c94eab42"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:23:27.221004 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:27.220907 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c92125c-50d9-4d98-9978-32c7c94eab42-dshm" pod "2c92125c-50d9-4d98-9978-32c7c94eab42" (UID: "2c92125c-50d9-4d98-9978-32c7c94eab42"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:23:27.221004 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:27.220907 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c92125c-50d9-4d98-9978-32c7c94eab42-kube-api-access-bdjjw" pod "2c92125c-50d9-4d98-9978-32c7c94eab42" (UID: "2c92125c-50d9-4d98-9978-32c7c94eab42"). InnerVolumeSpecName "kube-api-access-bdjjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:23:27.320103 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:27.320062 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdjjw\" (UniqueName: \"kubernetes.io/projected/2c92125c-50d9-4d98-9978-32c7c94eab42-kube-api-access-bdjjw\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:23:27.320103 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:27.320097 4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c92125c-50d9-4d98-9978-32c7c94eab42-tls-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:23:27.320103 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:27.320107 4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c92125c-50d9-4d98-9978-32c7c94eab42-dshm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:23:27.640910 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:27.640871 4931 generic.go:356] "Generic (PLEG): container finished" podID="2c92125c-50d9-4d98-9978-32c7c94eab42" containerID="9ba6972a468ecc349a21d58d857071d17571bf2de533a3297a72f3564b6114ed" exitCode=0 Oct 30 13:23:27.641085 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:27.640919 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb" Oct 30 13:23:27.641085 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:27.640964 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb" event={"ID":"2c92125c-50d9-4d98-9978-32c7c94eab42","Type":"ContainerDied","Data":"9ba6972a468ecc349a21d58d857071d17571bf2de533a3297a72f3564b6114ed"} Oct 30 13:23:27.641085 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:27.641027 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb" event={"ID":"2c92125c-50d9-4d98-9978-32c7c94eab42","Type":"ContainerDied","Data":"bb00b8e0d05a02c47bb78044d3936a3e0927457f34251a53299195829c246bd2"} Oct 30 13:23:27.641085 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:27.641050 4931 scope.go:122] "RemoveContainer" containerID="9ba6972a468ecc349a21d58d857071d17571bf2de533a3297a72f3564b6114ed" Oct 30 13:23:27.653963 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:27.653939 4931 scope.go:122] "RemoveContainer" containerID="9ba6972a468ecc349a21d58d857071d17571bf2de533a3297a72f3564b6114ed" Oct 30 13:23:27.654320 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:23:27.654259 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ba6972a468ecc349a21d58d857071d17571bf2de533a3297a72f3564b6114ed\": container with ID starting with 9ba6972a468ecc349a21d58d857071d17571bf2de533a3297a72f3564b6114ed not found: ID does not exist" containerID="9ba6972a468ecc349a21d58d857071d17571bf2de533a3297a72f3564b6114ed" Oct 30 13:23:27.654422 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:27.654324 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba6972a468ecc349a21d58d857071d17571bf2de533a3297a72f3564b6114ed"} err="failed to get container status \"9ba6972a468ecc349a21d58d857071d17571bf2de533a3297a72f3564b6114ed\": rpc error: code = NotFound desc = could not find container \"9ba6972a468ecc349a21d58d857071d17571bf2de533a3297a72f3564b6114ed\": container with ID starting with 9ba6972a468ecc349a21d58d857071d17571bf2de533a3297a72f3564b6114ed not found: ID does not exist" Oct 30 13:23:27.697877 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:27.697848 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb"] Oct 30 13:23:27.710819 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:27.710791 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-80fbd9/test-model-80fbd9-kserve-7f8bfcf4f7-2cxqb"] Oct 30 13:23:29.516018 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:23:29.515983 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c92125c-50d9-4d98-9978-32c7c94eab42" path="/var/lib/kubelet/pods/2c92125c-50d9-4d98-9978-32c7c94eab42/volumes" Oct 30 13:23:37.352898 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:23:37.352859 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:24:01.457040 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:01.457005 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j"] Oct 30 13:24:01.457697 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:01.457232 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j" podUID="59884998-e47d-448b-bf1d-9911a41b70c1" containerName="maas-api" containerID="cri-o://12fd1bb96ef84c5f4200ac70397f15b079e21c8d21dd5458f5e3776ec93a68d9" gracePeriod=30 Oct 30 13:24:01.750599 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:01.750571 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j" Oct 30 13:24:01.923100 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:01.915817 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59884998-e47d-448b-bf1d-9911a41b70c1-kube-api-access-qx8z6\" (UniqueName: \"kubernetes.io/projected/59884998-e47d-448b-bf1d-9911a41b70c1-kube-api-access-qx8z6\") pod \"59884998-e47d-448b-bf1d-9911a41b70c1\" (UID: \"59884998-e47d-448b-bf1d-9911a41b70c1\") " Oct 30 13:24:01.927086 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:01.927056 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59884998-e47d-448b-bf1d-9911a41b70c1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/59884998-e47d-448b-bf1d-9911a41b70c1-metrics-certs\") pod \"59884998-e47d-448b-bf1d-9911a41b70c1\" (UID: \"59884998-e47d-448b-bf1d-9911a41b70c1\") " Oct 30 13:24:01.927432 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:01.927409 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59884998-e47d-448b-bf1d-9911a41b70c1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59884998-e47d-448b-bf1d-9911a41b70c1-openshift-service-ca\") pod \"59884998-e47d-448b-bf1d-9911a41b70c1\" (UID: \"59884998-e47d-448b-bf1d-9911a41b70c1\") " Oct 30 13:24:01.927616 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:01.927597 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59884998-e47d-448b-bf1d-9911a41b70c1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59884998-e47d-448b-bf1d-9911a41b70c1-maas-api-tls\") pod \"59884998-e47d-448b-bf1d-9911a41b70c1\" (UID: \"59884998-e47d-448b-bf1d-9911a41b70c1\") " Oct 30 13:24:01.931715 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:01.931685 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59884998-e47d-448b-bf1d-9911a41b70c1-openshift-service-ca" pod "59884998-e47d-448b-bf1d-9911a41b70c1" (UID: "59884998-e47d-448b-bf1d-9911a41b70c1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:24:01.939936 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:01.939902 4931 generic.go:356] "Generic (PLEG): container finished" podID="59884998-e47d-448b-bf1d-9911a41b70c1" containerID="12fd1bb96ef84c5f4200ac70397f15b079e21c8d21dd5458f5e3776ec93a68d9" exitCode=0 Oct 30 13:24:01.940183 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:01.940164 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j" event={"ID":"59884998-e47d-448b-bf1d-9911a41b70c1","Type":"ContainerDied","Data":"12fd1bb96ef84c5f4200ac70397f15b079e21c8d21dd5458f5e3776ec93a68d9"} Oct 30 13:24:01.940257 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:01.940217 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j" event={"ID":"59884998-e47d-448b-bf1d-9911a41b70c1","Type":"ContainerDied","Data":"36183a5ff25f8f9381f6bbb49ba8dfe0d453c8ed7f732d4418ac4fa3167c0113"} Oct 30 13:24:01.940257 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:01.940247 4931 scope.go:122] "RemoveContainer" containerID="12fd1bb96ef84c5f4200ac70397f15b079e21c8d21dd5458f5e3776ec93a68d9" Oct 30 13:24:01.940435 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:01.940425 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j" Oct 30 13:24:01.950481 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:01.950431 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59884998-e47d-448b-bf1d-9911a41b70c1-metrics-certs" pod "59884998-e47d-448b-bf1d-9911a41b70c1" (UID: "59884998-e47d-448b-bf1d-9911a41b70c1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:24:01.950714 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:01.950676 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59884998-e47d-448b-bf1d-9911a41b70c1-maas-api-tls" pod "59884998-e47d-448b-bf1d-9911a41b70c1" (UID: "59884998-e47d-448b-bf1d-9911a41b70c1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:24:01.950975 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:01.950936 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59884998-e47d-448b-bf1d-9911a41b70c1-kube-api-access-qx8z6" pod "59884998-e47d-448b-bf1d-9911a41b70c1" (UID: "59884998-e47d-448b-bf1d-9911a41b70c1"). InnerVolumeSpecName "kube-api-access-qx8z6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:24:02.020425 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:01.993675 4931 scope.go:122] "RemoveContainer" containerID="12fd1bb96ef84c5f4200ac70397f15b079e21c8d21dd5458f5e3776ec93a68d9" Oct 30 13:24:02.020425 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:24:02.011857 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12fd1bb96ef84c5f4200ac70397f15b079e21c8d21dd5458f5e3776ec93a68d9\": container with ID starting with 12fd1bb96ef84c5f4200ac70397f15b079e21c8d21dd5458f5e3776ec93a68d9 not found: ID does not exist" containerID="12fd1bb96ef84c5f4200ac70397f15b079e21c8d21dd5458f5e3776ec93a68d9" Oct 30 13:24:02.020425 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:02.011906 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12fd1bb96ef84c5f4200ac70397f15b079e21c8d21dd5458f5e3776ec93a68d9"} err="failed to get container status \"12fd1bb96ef84c5f4200ac70397f15b079e21c8d21dd5458f5e3776ec93a68d9\": rpc error: code = NotFound desc = could not find container \"12fd1bb96ef84c5f4200ac70397f15b079e21c8d21dd5458f5e3776ec93a68d9\": container with ID starting with 12fd1bb96ef84c5f4200ac70397f15b079e21c8d21dd5458f5e3776ec93a68d9 not found: ID does not exist" Oct 30 13:24:02.031960 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:02.031835 4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59884998-e47d-448b-bf1d-9911a41b70c1-openshift-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:24:02.031960 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:02.031870 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59884998-e47d-448b-bf1d-9911a41b70c1-maas-api-tls\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:24:02.031960 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:02.031885 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx8z6\" (UniqueName: \"kubernetes.io/projected/59884998-e47d-448b-bf1d-9911a41b70c1-kube-api-access-qx8z6\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:24:02.031960 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:02.031897 4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59884998-e47d-448b-bf1d-9911a41b70c1-metrics-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:24:02.290673 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:02.290634 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j"] Oct 30 13:24:02.300576 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:02.300545 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cd2d2e-65fbb64dcd-64k2j"] Oct 30 13:24:03.523390 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:03.523355 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59884998-e47d-448b-bf1d-9911a41b70c1" path="/var/lib/kubelet/pods/59884998-e47d-448b-bf1d-9911a41b70c1/volumes" Oct 30 13:24:07.353044 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:24:07.352999 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:24:10.877711 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:10.877672 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:24:16.031866 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:16.031828 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77dd3317-nwgz6"] Oct 30 13:24:16.044822 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:16.044796 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77dd3317-nwgz6"] Oct 30 13:24:17.514421 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:17.514382 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dce7a7cd-c031-47f5-af51-5fdbe4f9e435" path="/var/lib/kubelet/pods/dce7a7cd-c031-47f5-af51-5fdbe4f9e435/volumes" Oct 30 13:24:37.352185 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:24:37.352149 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:24:37.626971 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.626888 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv"] Oct 30 13:24:37.627592 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.627573 4931 cpu_manager.go:414] "containerMap: removing container" podUID="1a9754b7-dc87-483f-9164-037a383d1f99" containerName="maas-api" Oct 30 13:24:37.627648 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.627594 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a9754b7-dc87-483f-9164-037a383d1f99" containerName="maas-api" Oct 30 13:24:37.627648 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.627612 4931 cpu_manager.go:414] "containerMap: removing container" podUID="7c31718e-7be5-4591-b32b-bf9655c06c2a" containerName="maas-api" Oct 30 13:24:37.627648 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.627618 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c31718e-7be5-4591-b32b-bf9655c06c2a" containerName="maas-api" Oct 30 13:24:37.627648 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.627629 4931 cpu_manager.go:414] "containerMap: removing container" podUID="59884998-e47d-448b-bf1d-9911a41b70c1" containerName="maas-api" Oct 30 13:24:37.627648 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.627634 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59884998-e47d-448b-bf1d-9911a41b70c1" containerName="maas-api" Oct 30 13:24:37.627648 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.627650 4931 cpu_manager.go:414] "containerMap: removing container" podUID="2023086a-0026-41a0-9180-d48b4523e392" containerName="main" Oct 30 13:24:37.627837 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.627656 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2023086a-0026-41a0-9180-d48b4523e392" containerName="main" Oct 30 13:24:37.627837 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.627671 4931 cpu_manager.go:414] "containerMap: removing container" podUID="01524a5a-15e5-45ad-a9cb-15b6d605bf86" containerName="istio-proxy" Oct 30 13:24:37.627837 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.627676 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01524a5a-15e5-45ad-a9cb-15b6d605bf86" containerName="istio-proxy" Oct 30 13:24:37.627837 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.627699 4931 cpu_manager.go:414] "containerMap: removing container" podUID="5c77e909-e679-4e88-8569-b406b4582f75" containerName="main" Oct 30 13:24:37.627837 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.627704 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c77e909-e679-4e88-8569-b406b4582f75" containerName="main" Oct 30 13:24:37.627837 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.627716 4931 cpu_manager.go:414] "containerMap: removing container" podUID="3eb1597f-ffc6-46f2-a6fb-34e000c3ef83" containerName="main" Oct 30 13:24:37.627837 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.627721 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eb1597f-ffc6-46f2-a6fb-34e000c3ef83" containerName="main" Oct 30 13:24:37.627837 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.627730 4931 cpu_manager.go:414] "containerMap: removing container" podUID="2c92125c-50d9-4d98-9978-32c7c94eab42" containerName="main" Oct 30 13:24:37.627837 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.627735 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c92125c-50d9-4d98-9978-32c7c94eab42" containerName="main" Oct 30 13:24:37.627837 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.627743 4931 cpu_manager.go:414] "containerMap: removing container" podUID="608feda4-9f34-43d8-b6bb-e3cc096f7d5c" containerName="istio-proxy" Oct 30 13:24:37.627837 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.627748 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="608feda4-9f34-43d8-b6bb-e3cc096f7d5c" containerName="istio-proxy" Oct 30 13:24:37.628152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.627845 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c77e909-e679-4e88-8569-b406b4582f75" containerName="main" Oct 30 13:24:37.628152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.627853 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eb1597f-ffc6-46f2-a6fb-34e000c3ef83" containerName="main" Oct 30 13:24:37.628152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.627860 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c92125c-50d9-4d98-9978-32c7c94eab42" containerName="main" Oct 30 13:24:37.628152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.627866 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="608feda4-9f34-43d8-b6bb-e3cc096f7d5c" containerName="istio-proxy" Oct 30 13:24:37.628152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.627879 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a9754b7-dc87-483f-9164-037a383d1f99" containerName="maas-api" Oct 30 13:24:37.628152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.627890 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c31718e-7be5-4591-b32b-bf9655c06c2a" containerName="maas-api" Oct 30 13:24:37.628152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.627897 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="59884998-e47d-448b-bf1d-9911a41b70c1" containerName="maas-api" Oct 30 13:24:37.628152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.627905 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="2023086a-0026-41a0-9180-d48b4523e392" containerName="main" Oct 30 13:24:37.628152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.627911 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="01524a5a-15e5-45ad-a9cb-15b6d605bf86" containerName="istio-proxy" Oct 30 13:24:37.631552 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.631530 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" Oct 30 13:24:37.650874 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.650832 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7b9784-abaf-439d-a30a-b19644f3c34a-credential-socket\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" Oct 30 13:24:37.651043 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.650939 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc7b9784-abaf-439d-a30a-b19644f3c34a-istiod-ca-cert\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" Oct 30 13:24:37.651043 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.651008 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc7b9784-abaf-439d-a30a-b19644f3c34a-istio-podinfo\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" Oct 30 13:24:37.651119 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.651049 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc7b9784-abaf-439d-a30a-b19644f3c34a-workload-certs\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" Oct 30 13:24:37.651119 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.651101 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc7b9784-abaf-439d-a30a-b19644f3c34a-istio-data\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" Oct 30 13:24:37.651188 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.651129 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc7b9784-abaf-439d-a30a-b19644f3c34a-istio-envoy\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" Oct 30 13:24:37.651225 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.651187 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9mbl\" (UniqueName: \"kubernetes.io/projected/dc7b9784-abaf-439d-a30a-b19644f3c34a-kube-api-access-k9mbl\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" Oct 30 13:24:37.651354 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.651339 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7b9784-abaf-439d-a30a-b19644f3c34a-workload-socket\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" Oct 30 13:24:37.651405 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.651394 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc7b9784-abaf-439d-a30a-b19644f3c34a-istio-token\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" Oct 30 13:24:37.662587 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.662554 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv"] Oct 30 13:24:37.671466 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.671434 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-0e874e-openshift-default-dockercfg-4czbc\"" Oct 30 13:24:37.752420 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.752380 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc7b9784-abaf-439d-a30a-b19644f3c34a-istio-token\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" Oct 30 13:24:37.752614 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.752465 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7b9784-abaf-439d-a30a-b19644f3c34a-credential-socket\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" Oct 30 13:24:37.752614 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.752515 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc7b9784-abaf-439d-a30a-b19644f3c34a-istiod-ca-cert\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" Oct 30 13:24:37.752614 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.752546 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc7b9784-abaf-439d-a30a-b19644f3c34a-istio-podinfo\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" Oct 30 13:24:37.752614 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.752582 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc7b9784-abaf-439d-a30a-b19644f3c34a-workload-certs\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" Oct 30 13:24:37.752614 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.752608 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc7b9784-abaf-439d-a30a-b19644f3c34a-istio-data\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" Oct 30 13:24:37.752883 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.752642 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc7b9784-abaf-439d-a30a-b19644f3c34a-istio-envoy\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" Oct 30 13:24:37.752883 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.752667 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9mbl\" (UniqueName: \"kubernetes.io/projected/dc7b9784-abaf-439d-a30a-b19644f3c34a-kube-api-access-k9mbl\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" Oct 30 13:24:37.752883 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.752760 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7b9784-abaf-439d-a30a-b19644f3c34a-workload-socket\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" Oct 30 13:24:37.753048 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.752985 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7b9784-abaf-439d-a30a-b19644f3c34a-credential-socket\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" Oct 30 13:24:37.753104 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.753064 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc7b9784-abaf-439d-a30a-b19644f3c34a-workload-certs\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" Oct 30 13:24:37.753173 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.753148 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc7b9784-abaf-439d-a30a-b19644f3c34a-istio-data\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" Oct 30 13:24:37.753345 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.753319 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc7b9784-abaf-439d-a30a-b19644f3c34a-istiod-ca-cert\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" Oct 30 13:24:37.753478 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.753425 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7b9784-abaf-439d-a30a-b19644f3c34a-workload-socket\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" Oct 30 13:24:37.755062 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.755046 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc7b9784-abaf-439d-a30a-b19644f3c34a-istio-envoy\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" Oct 30 13:24:37.755378 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.755361 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc7b9784-abaf-439d-a30a-b19644f3c34a-istio-podinfo\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" Oct 30 13:24:37.777383 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.777346 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc7b9784-abaf-439d-a30a-b19644f3c34a-istio-token\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" Oct 30 13:24:37.777517 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.777467 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9mbl\" (UniqueName: \"kubernetes.io/projected/dc7b9784-abaf-439d-a30a-b19644f3c34a-kube-api-access-k9mbl\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" Oct 30 13:24:37.947074 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:37.947036 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" Oct 30 13:24:38.272049 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:38.272018 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv"] Oct 30 13:24:38.272712 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:24:38.272681 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc7b9784_abaf_439d_a30a_b19644f3c34a.slice/crio-a749055da6ea057296d9cc35aa5686acdf6649d5d69fab910967c93c50b49906 WatchSource:0}: Error finding container a749055da6ea057296d9cc35aa5686acdf6649d5d69fab910967c93c50b49906: Status 404 returned error can't find the container with id a749055da6ea057296d9cc35aa5686acdf6649d5d69fab910967c93c50b49906 Oct 30 13:24:38.275353 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:38.275320 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:24:38.275424 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:38.275386 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:24:38.275424 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:38.275414 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:24:39.267347 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:39.267304 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" event={"ID":"dc7b9784-abaf-439d-a30a-b19644f3c34a","Type":"ContainerStarted","Data":"619d13d3f7a48c75421811ec74f0998f0f738efcc57c27da3af1ecdb21826952"} Oct 30 13:24:39.267876 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:39.267411 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" event={"ID":"dc7b9784-abaf-439d-a30a-b19644f3c34a","Type":"ContainerStarted","Data":"a749055da6ea057296d9cc35aa5686acdf6649d5d69fab910967c93c50b49906"} Oct 30 13:24:39.375195 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:39.375135 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" podStartSLOduration=2.375120586 podStartE2EDuration="2.375120586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:24:39.364755549 +0000 UTC m=+2377.968253994" watchObservedRunningTime="2026-10-30 13:24:39.375120586 +0000 UTC m=+2377.978619032" Oct 30 13:24:39.955287 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:39.951622 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" Oct 30 13:24:39.957398 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:39.957365 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 13:24:39.957564 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:39.957415 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" podUID="dc7b9784-abaf-439d-a30a-b19644f3c34a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 13:24:40.951546 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:40.951508 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 13:24:40.952980 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:40.952940 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" podUID="dc7b9784-abaf-439d-a30a-b19644f3c34a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 13:24:41.954723 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:41.954683 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 13:24:41.956381 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:41.956329 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" podUID="dc7b9784-abaf-439d-a30a-b19644f3c34a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 13:24:42.953346 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:42.952680 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 13:24:42.953346 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:42.952779 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" podUID="dc7b9784-abaf-439d-a30a-b19644f3c34a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 13:24:43.951297 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:43.947707 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 13:24:43.951297 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:43.947774 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" podUID="dc7b9784-abaf-439d-a30a-b19644f3c34a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 13:24:45.086909 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:45.086868 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" Oct 30 13:24:45.087479 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:45.087327 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" Oct 30 13:24:45.088428 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:24:45.088397 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" Oct 30 13:25:01.100326 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:01.100289 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp"] Oct 30 13:25:01.100743 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:01.100553 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp" podUID="e6b7f53a-3de6-4548-b270-48cb0a46030e" containerName="maas-api" containerID="cri-o://0f24429675f77e9fca8c767af98420305b85ab7bc468c4f1c80b2a604c81135d" gracePeriod=30 Oct 30 13:25:01.497945 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:01.497913 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp" Oct 30 13:25:01.498507 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:01.497978 4931 generic.go:356] "Generic (PLEG): container finished" podID="e6b7f53a-3de6-4548-b270-48cb0a46030e" containerID="0f24429675f77e9fca8c767af98420305b85ab7bc468c4f1c80b2a604c81135d" exitCode=0 Oct 30 13:25:01.498507 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:01.498000 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp" event={"ID":"e6b7f53a-3de6-4548-b270-48cb0a46030e","Type":"ContainerDied","Data":"0f24429675f77e9fca8c767af98420305b85ab7bc468c4f1c80b2a604c81135d"} Oct 30 13:25:01.498507 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:01.498368 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp" event={"ID":"e6b7f53a-3de6-4548-b270-48cb0a46030e","Type":"ContainerDied","Data":"b340493e94a88aaf51fc2c397ff614c48fe83ae9a19e4e6cca86c4f317e5c315"} Oct 30 13:25:01.498507 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:01.498402 4931 scope.go:122] "RemoveContainer" containerID="0f24429675f77e9fca8c767af98420305b85ab7bc468c4f1c80b2a604c81135d" Oct 30 13:25:01.515182 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:01.515088 4931 scope.go:122] "RemoveContainer" containerID="0f24429675f77e9fca8c767af98420305b85ab7bc468c4f1c80b2a604c81135d" Oct 30 13:25:01.515858 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:25:01.515698 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f24429675f77e9fca8c767af98420305b85ab7bc468c4f1c80b2a604c81135d\": container with ID starting with 0f24429675f77e9fca8c767af98420305b85ab7bc468c4f1c80b2a604c81135d not found: ID does not exist" containerID="0f24429675f77e9fca8c767af98420305b85ab7bc468c4f1c80b2a604c81135d" Oct 30 13:25:01.515858 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:01.515754 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f24429675f77e9fca8c767af98420305b85ab7bc468c4f1c80b2a604c81135d"} err="failed to get container status \"0f24429675f77e9fca8c767af98420305b85ab7bc468c4f1c80b2a604c81135d\": rpc error: code = NotFound desc = could not find container \"0f24429675f77e9fca8c767af98420305b85ab7bc468c4f1c80b2a604c81135d\": container with ID starting with 0f24429675f77e9fca8c767af98420305b85ab7bc468c4f1c80b2a604c81135d not found: ID does not exist" Oct 30 13:25:01.585288 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:01.574243 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6b7f53a-3de6-4548-b270-48cb0a46030e-kube-api-access-xkn47\" (UniqueName: \"kubernetes.io/projected/e6b7f53a-3de6-4548-b270-48cb0a46030e-kube-api-access-xkn47\") pod \"e6b7f53a-3de6-4548-b270-48cb0a46030e\" (UID: \"e6b7f53a-3de6-4548-b270-48cb0a46030e\") " Oct 30 13:25:01.585288 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:01.574324 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6b7f53a-3de6-4548-b270-48cb0a46030e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6b7f53a-3de6-4548-b270-48cb0a46030e-openshift-service-ca\") pod \"e6b7f53a-3de6-4548-b270-48cb0a46030e\" (UID: \"e6b7f53a-3de6-4548-b270-48cb0a46030e\") " Oct 30 13:25:01.585288 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:01.574385 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6b7f53a-3de6-4548-b270-48cb0a46030e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6b7f53a-3de6-4548-b270-48cb0a46030e-metrics-certs\") pod \"e6b7f53a-3de6-4548-b270-48cb0a46030e\" (UID: \"e6b7f53a-3de6-4548-b270-48cb0a46030e\") " Oct 30 13:25:01.585288 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:01.582996 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6b7f53a-3de6-4548-b270-48cb0a46030e-openshift-service-ca" pod "e6b7f53a-3de6-4548-b270-48cb0a46030e" (UID: "e6b7f53a-3de6-4548-b270-48cb0a46030e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:25:01.650037 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:01.649896 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6b7f53a-3de6-4548-b270-48cb0a46030e-metrics-certs" pod "e6b7f53a-3de6-4548-b270-48cb0a46030e" (UID: "e6b7f53a-3de6-4548-b270-48cb0a46030e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:25:01.679583 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:01.677346 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6b7f53a-3de6-4548-b270-48cb0a46030e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6b7f53a-3de6-4548-b270-48cb0a46030e-maas-api-tls\") pod \"e6b7f53a-3de6-4548-b270-48cb0a46030e\" (UID: \"e6b7f53a-3de6-4548-b270-48cb0a46030e\") " Oct 30 13:25:01.679583 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:01.678105 4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6b7f53a-3de6-4548-b270-48cb0a46030e-openshift-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:25:01.679583 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:01.678128 4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6b7f53a-3de6-4548-b270-48cb0a46030e-metrics-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:25:01.742333 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:01.740585 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6b7f53a-3de6-4548-b270-48cb0a46030e-kube-api-access-xkn47" pod "e6b7f53a-3de6-4548-b270-48cb0a46030e" (UID: "e6b7f53a-3de6-4548-b270-48cb0a46030e"). InnerVolumeSpecName "kube-api-access-xkn47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:01.749200 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:01.749151 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6b7f53a-3de6-4548-b270-48cb0a46030e-maas-api-tls" pod "e6b7f53a-3de6-4548-b270-48cb0a46030e" (UID: "e6b7f53a-3de6-4548-b270-48cb0a46030e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:25:01.779830 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:01.779794 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6b7f53a-3de6-4548-b270-48cb0a46030e-maas-api-tls\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:25:01.779830 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:01.779828 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkn47\" (UniqueName: \"kubernetes.io/projected/e6b7f53a-3de6-4548-b270-48cb0a46030e-kube-api-access-xkn47\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:25:01.983726 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:01.983696 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/2.log" Oct 30 13:25:02.000641 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:02.000609 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/2.log" Oct 30 13:25:02.506436 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:02.506405 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp" Oct 30 13:25:02.562506 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:02.562479 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp"] Oct 30 13:25:02.567773 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:02.567749 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-618c72-5cc4b8898d-7v9xp"] Oct 30 13:25:03.515099 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:03.515009 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6b7f53a-3de6-4548-b270-48cb0a46030e" path="/var/lib/kubelet/pods/e6b7f53a-3de6-4548-b270-48cb0a46030e/volumes" Oct 30 13:25:07.351800 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:25:07.351765 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:25:11.340292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:11.339429 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cc9ad88-nc5m4"] Oct 30 13:25:11.340292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:11.340244 4931 cpu_manager.go:414] "containerMap: removing container" podUID="e6b7f53a-3de6-4548-b270-48cb0a46030e" containerName="maas-api" Oct 30 13:25:11.340292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:11.340283 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6b7f53a-3de6-4548-b270-48cb0a46030e" containerName="maas-api" Oct 30 13:25:11.340959 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:11.340437 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6b7f53a-3de6-4548-b270-48cb0a46030e" containerName="maas-api" Oct 30 13:25:11.349292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:11.346042 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc9ad88-nc5m4" Oct 30 13:25:11.370882 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:11.370851 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:25:11.387878 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:11.387837 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:25:11.407305 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:11.407254 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cc9ad88-nc5m4"] Oct 30 13:25:11.460387 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:11.459722 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e946299e-2bd0-4ad8-af2c-6fc3340fd967-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cc9ad88-nc5m4\" (UID: \"e946299e-2bd0-4ad8-af2c-6fc3340fd967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc9ad88-nc5m4" Oct 30 13:25:11.460387 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:11.459939 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e946299e-2bd0-4ad8-af2c-6fc3340fd967-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cc9ad88-nc5m4\" (UID: \"e946299e-2bd0-4ad8-af2c-6fc3340fd967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc9ad88-nc5m4" Oct 30 13:25:11.460387 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:11.460118 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnrnc\" (UniqueName: \"kubernetes.io/projected/e946299e-2bd0-4ad8-af2c-6fc3340fd967-kube-api-access-tnrnc\") pod \"maas-api-revoke-sub-0cc9ad88-nc5m4\" (UID: \"e946299e-2bd0-4ad8-af2c-6fc3340fd967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc9ad88-nc5m4" Oct 30 13:25:11.561184 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:11.561146 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnrnc\" (UniqueName: \"kubernetes.io/projected/e946299e-2bd0-4ad8-af2c-6fc3340fd967-kube-api-access-tnrnc\") pod \"maas-api-revoke-sub-0cc9ad88-nc5m4\" (UID: \"e946299e-2bd0-4ad8-af2c-6fc3340fd967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc9ad88-nc5m4" Oct 30 13:25:11.561419 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:11.561227 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e946299e-2bd0-4ad8-af2c-6fc3340fd967-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cc9ad88-nc5m4\" (UID: \"e946299e-2bd0-4ad8-af2c-6fc3340fd967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc9ad88-nc5m4" Oct 30 13:25:11.561419 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:11.561384 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e946299e-2bd0-4ad8-af2c-6fc3340fd967-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cc9ad88-nc5m4\" (UID: \"e946299e-2bd0-4ad8-af2c-6fc3340fd967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc9ad88-nc5m4" Oct 30 13:25:11.562089 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:11.562058 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e946299e-2bd0-4ad8-af2c-6fc3340fd967-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cc9ad88-nc5m4\" (UID: \"e946299e-2bd0-4ad8-af2c-6fc3340fd967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc9ad88-nc5m4" Oct 30 13:25:11.563694 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:11.563670 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e946299e-2bd0-4ad8-af2c-6fc3340fd967-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cc9ad88-nc5m4\" (UID: \"e946299e-2bd0-4ad8-af2c-6fc3340fd967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc9ad88-nc5m4" Oct 30 13:25:11.586634 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:11.586604 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnrnc\" (UniqueName: \"kubernetes.io/projected/e946299e-2bd0-4ad8-af2c-6fc3340fd967-kube-api-access-tnrnc\") pod \"maas-api-revoke-sub-0cc9ad88-nc5m4\" (UID: \"e946299e-2bd0-4ad8-af2c-6fc3340fd967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc9ad88-nc5m4" Oct 30 13:25:11.681535 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:11.681452 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc9ad88-nc5m4" Oct 30 13:25:11.995111 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:11.995077 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cc9ad88-nc5m4"] Oct 30 13:25:11.997599 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:25:11.997567 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode946299e_2bd0_4ad8_af2c_6fc3340fd967.slice/crio-77d8ad73857667d978aa8dd5386895c8dec9695680ec7d49e80cacc476e7306f WatchSource:0}: Error finding container 77d8ad73857667d978aa8dd5386895c8dec9695680ec7d49e80cacc476e7306f: Status 404 returned error can't find the container with id 77d8ad73857667d978aa8dd5386895c8dec9695680ec7d49e80cacc476e7306f Oct 30 13:25:12.579539 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:12.579451 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc9ad88-nc5m4" event={"ID":"e946299e-2bd0-4ad8-af2c-6fc3340fd967","Type":"ContainerStarted","Data":"be65020a256f468edba8d916c5a57b71b02ac388176ab33b84992c4e0346b2ee"} Oct 30 13:25:12.579539 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:12.579518 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc9ad88-nc5m4" event={"ID":"e946299e-2bd0-4ad8-af2c-6fc3340fd967","Type":"ContainerStarted","Data":"77d8ad73857667d978aa8dd5386895c8dec9695680ec7d49e80cacc476e7306f"} Oct 30 13:25:12.635748 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:12.635687 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc9ad88-nc5m4" podStartSLOduration=1.635673143 podStartE2EDuration="1.635673143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:12.634474242 +0000 UTC m=+2411.237972689" watchObservedRunningTime="2026-10-30 13:25:12.635673143 +0000 UTC m=+2411.239171588" Oct 30 13:25:13.587941 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:13.587908 4931 generic.go:356] "Generic (PLEG): container finished" podID="e946299e-2bd0-4ad8-af2c-6fc3340fd967" containerID="be65020a256f468edba8d916c5a57b71b02ac388176ab33b84992c4e0346b2ee" exitCode=0 Oct 30 13:25:13.588432 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:13.587992 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc9ad88-nc5m4" event={"ID":"e946299e-2bd0-4ad8-af2c-6fc3340fd967","Type":"ContainerDied","Data":"be65020a256f468edba8d916c5a57b71b02ac388176ab33b84992c4e0346b2ee"} Oct 30 13:25:14.763855 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:14.763826 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc9ad88-nc5m4" Oct 30 13:25:14.906238 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:14.906149 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e946299e-2bd0-4ad8-af2c-6fc3340fd967-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e946299e-2bd0-4ad8-af2c-6fc3340fd967-maas-api-openshift-ca\") pod \"e946299e-2bd0-4ad8-af2c-6fc3340fd967\" (UID: \"e946299e-2bd0-4ad8-af2c-6fc3340fd967\") " Oct 30 13:25:14.906457 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:14.906339 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e946299e-2bd0-4ad8-af2c-6fc3340fd967-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e946299e-2bd0-4ad8-af2c-6fc3340fd967-maas-api-xks-ca\") pod \"e946299e-2bd0-4ad8-af2c-6fc3340fd967\" (UID: \"e946299e-2bd0-4ad8-af2c-6fc3340fd967\") " Oct 30 13:25:14.906530 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:14.906456 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e946299e-2bd0-4ad8-af2c-6fc3340fd967-kube-api-access-tnrnc\" (UniqueName: \"kubernetes.io/projected/e946299e-2bd0-4ad8-af2c-6fc3340fd967-kube-api-access-tnrnc\") pod \"e946299e-2bd0-4ad8-af2c-6fc3340fd967\" (UID: \"e946299e-2bd0-4ad8-af2c-6fc3340fd967\") " Oct 30 13:25:14.906585 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:14.906548 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e946299e-2bd0-4ad8-af2c-6fc3340fd967-maas-api-openshift-ca" pod "e946299e-2bd0-4ad8-af2c-6fc3340fd967" (UID: "e946299e-2bd0-4ad8-af2c-6fc3340fd967"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:25:14.906939 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:14.906921 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e946299e-2bd0-4ad8-af2c-6fc3340fd967-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:25:14.908647 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:14.908626 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e946299e-2bd0-4ad8-af2c-6fc3340fd967-kube-api-access-tnrnc" pod "e946299e-2bd0-4ad8-af2c-6fc3340fd967" (UID: "e946299e-2bd0-4ad8-af2c-6fc3340fd967"). InnerVolumeSpecName "kube-api-access-tnrnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:14.908719 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:14.908693 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e946299e-2bd0-4ad8-af2c-6fc3340fd967-maas-api-xks-ca" pod "e946299e-2bd0-4ad8-af2c-6fc3340fd967" (UID: "e946299e-2bd0-4ad8-af2c-6fc3340fd967"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:25:15.007954 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:15.007916 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e946299e-2bd0-4ad8-af2c-6fc3340fd967-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:25:15.007954 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:15.007950 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnrnc\" (UniqueName: \"kubernetes.io/projected/e946299e-2bd0-4ad8-af2c-6fc3340fd967-kube-api-access-tnrnc\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:25:15.089018 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:15.088986 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w"] Oct 30 13:25:15.089231 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:15.089203 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" podUID="3dd90613-12ac-4509-9f2e-b5c1e199719f" containerName="istio-proxy" containerID="cri-o://a3844c4b0fed5f26e5fdb8fa7c78ad4c6c54b52251504aca93960d762f3377d5" gracePeriod=30 Oct 30 13:25:15.602021 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:15.601985 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc9ad88-nc5m4" event={"ID":"e946299e-2bd0-4ad8-af2c-6fc3340fd967","Type":"ContainerDied","Data":"77d8ad73857667d978aa8dd5386895c8dec9695680ec7d49e80cacc476e7306f"} Oct 30 13:25:15.602021 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:15.602008 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc9ad88-nc5m4" Oct 30 13:25:15.602227 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:15.602032 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77d8ad73857667d978aa8dd5386895c8dec9695680ec7d49e80cacc476e7306f" Oct 30 13:25:18.257703 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:18.257666 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-tvm7w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:25:18.258131 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:18.257739 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" podUID="3dd90613-12ac-4509-9f2e-b5c1e199719f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:25:20.375926 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.375829 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" Oct 30 13:25:20.488856 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.488821 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dd90613-12ac-4509-9f2e-b5c1e199719f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dd90613-12ac-4509-9f2e-b5c1e199719f-workload-certs\") pod \"3dd90613-12ac-4509-9f2e-b5c1e199719f\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " Oct 30 13:25:20.489009 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.488875 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dd90613-12ac-4509-9f2e-b5c1e199719f-kube-api-access-lgfbv\" (UniqueName: \"kubernetes.io/projected/3dd90613-12ac-4509-9f2e-b5c1e199719f-kube-api-access-lgfbv\") pod \"3dd90613-12ac-4509-9f2e-b5c1e199719f\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " Oct 30 13:25:20.489009 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.488939 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3dd90613-12ac-4509-9f2e-b5c1e199719f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dd90613-12ac-4509-9f2e-b5c1e199719f-istio-podinfo\") pod \"3dd90613-12ac-4509-9f2e-b5c1e199719f\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " Oct 30 13:25:20.489009 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.488978 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dd90613-12ac-4509-9f2e-b5c1e199719f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dd90613-12ac-4509-9f2e-b5c1e199719f-istiod-ca-cert\") pod \"3dd90613-12ac-4509-9f2e-b5c1e199719f\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " Oct 30 13:25:20.489188 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.489036 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dd90613-12ac-4509-9f2e-b5c1e199719f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dd90613-12ac-4509-9f2e-b5c1e199719f-credential-socket\") pod \"3dd90613-12ac-4509-9f2e-b5c1e199719f\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " Oct 30 13:25:20.489188 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.489094 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dd90613-12ac-4509-9f2e-b5c1e199719f-istio-token\" (UniqueName: \"kubernetes.io/projected/3dd90613-12ac-4509-9f2e-b5c1e199719f-istio-token\") pod \"3dd90613-12ac-4509-9f2e-b5c1e199719f\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " Oct 30 13:25:20.489188 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.489095 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dd90613-12ac-4509-9f2e-b5c1e199719f-workload-certs" pod "3dd90613-12ac-4509-9f2e-b5c1e199719f" (UID: "3dd90613-12ac-4509-9f2e-b5c1e199719f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:20.489188 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.489133 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dd90613-12ac-4509-9f2e-b5c1e199719f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dd90613-12ac-4509-9f2e-b5c1e199719f-workload-socket\") pod \"3dd90613-12ac-4509-9f2e-b5c1e199719f\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " Oct 30 13:25:20.489452 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.489255 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dd90613-12ac-4509-9f2e-b5c1e199719f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dd90613-12ac-4509-9f2e-b5c1e199719f-istio-data\") pod \"3dd90613-12ac-4509-9f2e-b5c1e199719f\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " Oct 30 13:25:20.489452 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.489339 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dd90613-12ac-4509-9f2e-b5c1e199719f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dd90613-12ac-4509-9f2e-b5c1e199719f-istio-envoy\") pod \"3dd90613-12ac-4509-9f2e-b5c1e199719f\" (UID: \"3dd90613-12ac-4509-9f2e-b5c1e199719f\") " Oct 30 13:25:20.489452 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.489439 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dd90613-12ac-4509-9f2e-b5c1e199719f-credential-socket" pod "3dd90613-12ac-4509-9f2e-b5c1e199719f" (UID: "3dd90613-12ac-4509-9f2e-b5c1e199719f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:20.489615 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.489509 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dd90613-12ac-4509-9f2e-b5c1e199719f-workload-socket" pod "3dd90613-12ac-4509-9f2e-b5c1e199719f" (UID: "3dd90613-12ac-4509-9f2e-b5c1e199719f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:20.489615 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.489527 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dd90613-12ac-4509-9f2e-b5c1e199719f-istiod-ca-cert" pod "3dd90613-12ac-4509-9f2e-b5c1e199719f" (UID: "3dd90613-12ac-4509-9f2e-b5c1e199719f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:25:20.489731 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.489626 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dd90613-12ac-4509-9f2e-b5c1e199719f-istio-data" pod "3dd90613-12ac-4509-9f2e-b5c1e199719f" (UID: "3dd90613-12ac-4509-9f2e-b5c1e199719f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:20.490019 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.489991 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dd90613-12ac-4509-9f2e-b5c1e199719f-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:25:20.490019 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.490021 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dd90613-12ac-4509-9f2e-b5c1e199719f-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:25:20.490237 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.490037 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dd90613-12ac-4509-9f2e-b5c1e199719f-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:25:20.490237 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.490053 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dd90613-12ac-4509-9f2e-b5c1e199719f-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:25:20.490237 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.490067 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dd90613-12ac-4509-9f2e-b5c1e199719f-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:25:20.491631 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.491602 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dd90613-12ac-4509-9f2e-b5c1e199719f-kube-api-access-lgfbv" pod "3dd90613-12ac-4509-9f2e-b5c1e199719f" (UID: "3dd90613-12ac-4509-9f2e-b5c1e199719f"). InnerVolumeSpecName "kube-api-access-lgfbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:20.491745 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.491630 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dd90613-12ac-4509-9f2e-b5c1e199719f-istio-token" pod "3dd90613-12ac-4509-9f2e-b5c1e199719f" (UID: "3dd90613-12ac-4509-9f2e-b5c1e199719f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:20.491882 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.491858 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3dd90613-12ac-4509-9f2e-b5c1e199719f-istio-podinfo" pod "3dd90613-12ac-4509-9f2e-b5c1e199719f" (UID: "3dd90613-12ac-4509-9f2e-b5c1e199719f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:25:20.491997 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.491962 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dd90613-12ac-4509-9f2e-b5c1e199719f-istio-envoy" pod "3dd90613-12ac-4509-9f2e-b5c1e199719f" (UID: "3dd90613-12ac-4509-9f2e-b5c1e199719f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:20.591248 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.591209 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dd90613-12ac-4509-9f2e-b5c1e199719f-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:25:20.591248 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.591244 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dd90613-12ac-4509-9f2e-b5c1e199719f-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:25:20.591248 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.591256 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dd90613-12ac-4509-9f2e-b5c1e199719f-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:25:20.591248 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.591278 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgfbv\" (UniqueName: \"kubernetes.io/projected/3dd90613-12ac-4509-9f2e-b5c1e199719f-kube-api-access-lgfbv\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:25:20.659031 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.658951 4931 generic.go:356] "Generic (PLEG): container finished" podID="3dd90613-12ac-4509-9f2e-b5c1e199719f" containerID="a3844c4b0fed5f26e5fdb8fa7c78ad4c6c54b52251504aca93960d762f3377d5" exitCode=0 Oct 30 13:25:20.659031 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.659001 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" Oct 30 13:25:20.659031 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.659017 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" event={"ID":"3dd90613-12ac-4509-9f2e-b5c1e199719f","Type":"ContainerDied","Data":"a3844c4b0fed5f26e5fdb8fa7c78ad4c6c54b52251504aca93960d762f3377d5"} Oct 30 13:25:20.659252 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.659085 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w" event={"ID":"3dd90613-12ac-4509-9f2e-b5c1e199719f","Type":"ContainerDied","Data":"97ce0758162b098bdddfc83ce422da0e6b2d6fd38f6c7ce19987153ce7e92988"} Oct 30 13:25:20.659252 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.659111 4931 scope.go:122] "RemoveContainer" containerID="a3844c4b0fed5f26e5fdb8fa7c78ad4c6c54b52251504aca93960d762f3377d5" Oct 30 13:25:20.670664 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.670648 4931 scope.go:122] "RemoveContainer" containerID="a3844c4b0fed5f26e5fdb8fa7c78ad4c6c54b52251504aca93960d762f3377d5" Oct 30 13:25:20.670935 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:25:20.670915 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3844c4b0fed5f26e5fdb8fa7c78ad4c6c54b52251504aca93960d762f3377d5\": container with ID starting with a3844c4b0fed5f26e5fdb8fa7c78ad4c6c54b52251504aca93960d762f3377d5 not found: ID does not exist" containerID="a3844c4b0fed5f26e5fdb8fa7c78ad4c6c54b52251504aca93960d762f3377d5" Oct 30 13:25:20.670979 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.670945 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3844c4b0fed5f26e5fdb8fa7c78ad4c6c54b52251504aca93960d762f3377d5"} err="failed to get container status \"a3844c4b0fed5f26e5fdb8fa7c78ad4c6c54b52251504aca93960d762f3377d5\": rpc error: code = NotFound desc = could not find container \"a3844c4b0fed5f26e5fdb8fa7c78ad4c6c54b52251504aca93960d762f3377d5\": container with ID starting with a3844c4b0fed5f26e5fdb8fa7c78ad4c6c54b52251504aca93960d762f3377d5 not found: ID does not exist" Oct 30 13:25:20.722459 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.722424 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w"] Oct 30 13:25:20.727414 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:20.727390 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w"] Oct 30 13:25:21.514868 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:21.514835 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dd90613-12ac-4509-9f2e-b5c1e199719f" path="/var/lib/kubelet/pods/3dd90613-12ac-4509-9f2e-b5c1e199719f/volumes" Oct 30 13:25:30.705586 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:30.704505 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:25:37.352715 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:25:37.352675 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:25:39.307082 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:39.307047 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84c737ea-7dbkv"] Oct 30 13:25:39.307631 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:39.307615 4931 cpu_manager.go:414] "containerMap: removing container" podUID="e946299e-2bd0-4ad8-af2c-6fc3340fd967" containerName="revoke-keys" Oct 30 13:25:39.307676 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:39.307632 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e946299e-2bd0-4ad8-af2c-6fc3340fd967" containerName="revoke-keys" Oct 30 13:25:39.307709 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:39.307680 4931 cpu_manager.go:414] "containerMap: removing container" podUID="3dd90613-12ac-4509-9f2e-b5c1e199719f" containerName="istio-proxy" Oct 30 13:25:39.307709 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:39.307686 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dd90613-12ac-4509-9f2e-b5c1e199719f" containerName="istio-proxy" Oct 30 13:25:39.307782 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:39.307772 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="e946299e-2bd0-4ad8-af2c-6fc3340fd967" containerName="revoke-keys" Oct 30 13:25:39.307821 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:39.307793 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dd90613-12ac-4509-9f2e-b5c1e199719f" containerName="istio-proxy" Oct 30 13:25:39.311004 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:39.310986 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c737ea-7dbkv" Oct 30 13:25:39.333778 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:39.333749 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84c737ea-7dbkv"] Oct 30 13:25:39.342070 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:39.342046 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:25:39.342262 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:39.342245 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:25:39.415535 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:39.415505 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ba7ef30-326a-48dd-b501-940c52c0c380-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84c737ea-7dbkv\" (UID: \"9ba7ef30-326a-48dd-b501-940c52c0c380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c737ea-7dbkv" Oct 30 13:25:39.415673 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:39.415557 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ba7ef30-326a-48dd-b501-940c52c0c380-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84c737ea-7dbkv\" (UID: \"9ba7ef30-326a-48dd-b501-940c52c0c380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c737ea-7dbkv" Oct 30 13:25:39.415673 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:39.415621 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s24ft\" (UniqueName: \"kubernetes.io/projected/9ba7ef30-326a-48dd-b501-940c52c0c380-kube-api-access-s24ft\") pod \"maas-api-revoke-sub-84c737ea-7dbkv\" (UID: \"9ba7ef30-326a-48dd-b501-940c52c0c380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c737ea-7dbkv" Oct 30 13:25:39.517010 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:39.516977 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ba7ef30-326a-48dd-b501-940c52c0c380-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84c737ea-7dbkv\" (UID: \"9ba7ef30-326a-48dd-b501-940c52c0c380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c737ea-7dbkv" Oct 30 13:25:39.517195 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:39.517035 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ba7ef30-326a-48dd-b501-940c52c0c380-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84c737ea-7dbkv\" (UID: \"9ba7ef30-326a-48dd-b501-940c52c0c380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c737ea-7dbkv" Oct 30 13:25:39.517195 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:39.517062 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s24ft\" (UniqueName: \"kubernetes.io/projected/9ba7ef30-326a-48dd-b501-940c52c0c380-kube-api-access-s24ft\") pod \"maas-api-revoke-sub-84c737ea-7dbkv\" (UID: \"9ba7ef30-326a-48dd-b501-940c52c0c380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c737ea-7dbkv" Oct 30 13:25:39.517668 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:39.517643 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ba7ef30-326a-48dd-b501-940c52c0c380-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84c737ea-7dbkv\" (UID: \"9ba7ef30-326a-48dd-b501-940c52c0c380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c737ea-7dbkv" Oct 30 13:25:39.519608 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:39.519590 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ba7ef30-326a-48dd-b501-940c52c0c380-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84c737ea-7dbkv\" (UID: \"9ba7ef30-326a-48dd-b501-940c52c0c380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c737ea-7dbkv" Oct 30 13:25:39.542114 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:39.542087 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s24ft\" (UniqueName: \"kubernetes.io/projected/9ba7ef30-326a-48dd-b501-940c52c0c380-kube-api-access-s24ft\") pod \"maas-api-revoke-sub-84c737ea-7dbkv\" (UID: \"9ba7ef30-326a-48dd-b501-940c52c0c380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c737ea-7dbkv" Oct 30 13:25:39.625168 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:39.625074 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c737ea-7dbkv" Oct 30 13:25:39.935478 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:39.935450 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84c737ea-7dbkv"] Oct 30 13:25:39.936553 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:25:39.936520 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ba7ef30_326a_48dd_b501_940c52c0c380.slice/crio-ccd0a0b17dba5b40e0d0b074e4859ffebfea1ed5790f0ac526fb0ed57a50c03c WatchSource:0}: Error finding container ccd0a0b17dba5b40e0d0b074e4859ffebfea1ed5790f0ac526fb0ed57a50c03c: Status 404 returned error can't find the container with id ccd0a0b17dba5b40e0d0b074e4859ffebfea1ed5790f0ac526fb0ed57a50c03c Oct 30 13:25:40.899720 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:40.899688 4931 generic.go:356] "Generic (PLEG): container finished" podID="9ba7ef30-326a-48dd-b501-940c52c0c380" containerID="34728b82ad869405b788a8202216648475e62adda843252179e01cacdcce1d30" exitCode=0 Oct 30 13:25:40.900085 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:40.899779 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c737ea-7dbkv" event={"ID":"9ba7ef30-326a-48dd-b501-940c52c0c380","Type":"ContainerDied","Data":"34728b82ad869405b788a8202216648475e62adda843252179e01cacdcce1d30"} Oct 30 13:25:40.900085 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:40.899833 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c737ea-7dbkv" event={"ID":"9ba7ef30-326a-48dd-b501-940c52c0c380","Type":"ContainerStarted","Data":"ccd0a0b17dba5b40e0d0b074e4859ffebfea1ed5790f0ac526fb0ed57a50c03c"} Oct 30 13:25:42.079069 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:42.079032 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c737ea-7dbkv" Oct 30 13:25:42.257152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:42.257115 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ba7ef30-326a-48dd-b501-940c52c0c380-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ba7ef30-326a-48dd-b501-940c52c0c380-maas-api-xks-ca\") pod \"9ba7ef30-326a-48dd-b501-940c52c0c380\" (UID: \"9ba7ef30-326a-48dd-b501-940c52c0c380\") " Oct 30 13:25:42.257392 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:42.257192 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ba7ef30-326a-48dd-b501-940c52c0c380-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ba7ef30-326a-48dd-b501-940c52c0c380-maas-api-openshift-ca\") pod \"9ba7ef30-326a-48dd-b501-940c52c0c380\" (UID: \"9ba7ef30-326a-48dd-b501-940c52c0c380\") " Oct 30 13:25:42.257467 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:42.257400 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ba7ef30-326a-48dd-b501-940c52c0c380-kube-api-access-s24ft\" (UniqueName: \"kubernetes.io/projected/9ba7ef30-326a-48dd-b501-940c52c0c380-kube-api-access-s24ft\") pod \"9ba7ef30-326a-48dd-b501-940c52c0c380\" (UID: \"9ba7ef30-326a-48dd-b501-940c52c0c380\") " Oct 30 13:25:42.257644 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:42.257601 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ba7ef30-326a-48dd-b501-940c52c0c380-maas-api-openshift-ca" pod "9ba7ef30-326a-48dd-b501-940c52c0c380" (UID: "9ba7ef30-326a-48dd-b501-940c52c0c380"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:25:42.258145 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:42.258116 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ba7ef30-326a-48dd-b501-940c52c0c380-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:25:42.259704 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:42.259676 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ba7ef30-326a-48dd-b501-940c52c0c380-kube-api-access-s24ft" pod "9ba7ef30-326a-48dd-b501-940c52c0c380" (UID: "9ba7ef30-326a-48dd-b501-940c52c0c380"). InnerVolumeSpecName "kube-api-access-s24ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:42.259819 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:42.259697 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ba7ef30-326a-48dd-b501-940c52c0c380-maas-api-xks-ca" pod "9ba7ef30-326a-48dd-b501-940c52c0c380" (UID: "9ba7ef30-326a-48dd-b501-940c52c0c380"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:25:42.359035 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:42.358996 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ba7ef30-326a-48dd-b501-940c52c0c380-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:25:42.359035 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:42.359031 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s24ft\" (UniqueName: \"kubernetes.io/projected/9ba7ef30-326a-48dd-b501-940c52c0c380-kube-api-access-s24ft\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:25:42.940300 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:42.939753 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c737ea-7dbkv" event={"ID":"9ba7ef30-326a-48dd-b501-940c52c0c380","Type":"ContainerDied","Data":"ccd0a0b17dba5b40e0d0b074e4859ffebfea1ed5790f0ac526fb0ed57a50c03c"} Oct 30 13:25:42.940300 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:42.939880 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccd0a0b17dba5b40e0d0b074e4859ffebfea1ed5790f0ac526fb0ed57a50c03c" Oct 30 13:25:42.940300 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:42.939961 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c737ea-7dbkv" Oct 30 13:25:51.951774 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:51.951736 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv"] Oct 30 13:25:51.952413 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:51.952002 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv" podUID="441dfaf9-d63b-4882-a70b-f1769a32ca9d" containerName="maas-api" containerID="cri-o://312305c9682e71a54ccd183a2455f6906ac79748c0b0645f794192b271291920" gracePeriod=30 Oct 30 13:25:52.323291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:52.317395 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv" Oct 30 13:25:52.414290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:52.413707 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/441dfaf9-d63b-4882-a70b-f1769a32ca9d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/441dfaf9-d63b-4882-a70b-f1769a32ca9d-openshift-service-ca\") pod \"441dfaf9-d63b-4882-a70b-f1769a32ca9d\" (UID: \"441dfaf9-d63b-4882-a70b-f1769a32ca9d\") " Oct 30 13:25:52.414290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:52.413775 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/441dfaf9-d63b-4882-a70b-f1769a32ca9d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/441dfaf9-d63b-4882-a70b-f1769a32ca9d-maas-api-tls\") pod \"441dfaf9-d63b-4882-a70b-f1769a32ca9d\" (UID: \"441dfaf9-d63b-4882-a70b-f1769a32ca9d\") " Oct 30 13:25:52.414290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:52.413815 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/441dfaf9-d63b-4882-a70b-f1769a32ca9d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/441dfaf9-d63b-4882-a70b-f1769a32ca9d-metrics-certs\") pod \"441dfaf9-d63b-4882-a70b-f1769a32ca9d\" (UID: \"441dfaf9-d63b-4882-a70b-f1769a32ca9d\") " Oct 30 13:25:52.414290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:52.413902 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/441dfaf9-d63b-4882-a70b-f1769a32ca9d-kube-api-access-wkk5h\" (UniqueName: \"kubernetes.io/projected/441dfaf9-d63b-4882-a70b-f1769a32ca9d-kube-api-access-wkk5h\") pod \"441dfaf9-d63b-4882-a70b-f1769a32ca9d\" (UID: \"441dfaf9-d63b-4882-a70b-f1769a32ca9d\") " Oct 30 13:25:52.414290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:52.414172 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/441dfaf9-d63b-4882-a70b-f1769a32ca9d-openshift-service-ca" pod "441dfaf9-d63b-4882-a70b-f1769a32ca9d" (UID: "441dfaf9-d63b-4882-a70b-f1769a32ca9d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:25:52.414779 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:52.414624 4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/441dfaf9-d63b-4882-a70b-f1769a32ca9d-openshift-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:25:52.418245 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:52.417926 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/441dfaf9-d63b-4882-a70b-f1769a32ca9d-maas-api-tls" pod "441dfaf9-d63b-4882-a70b-f1769a32ca9d" (UID: "441dfaf9-d63b-4882-a70b-f1769a32ca9d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:25:52.418425 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:52.418388 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/441dfaf9-d63b-4882-a70b-f1769a32ca9d-kube-api-access-wkk5h" pod "441dfaf9-d63b-4882-a70b-f1769a32ca9d" (UID: "441dfaf9-d63b-4882-a70b-f1769a32ca9d"). InnerVolumeSpecName "kube-api-access-wkk5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:52.418741 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:52.418717 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/441dfaf9-d63b-4882-a70b-f1769a32ca9d-metrics-certs" pod "441dfaf9-d63b-4882-a70b-f1769a32ca9d" (UID: "441dfaf9-d63b-4882-a70b-f1769a32ca9d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:25:52.522344 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:52.519232 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/441dfaf9-d63b-4882-a70b-f1769a32ca9d-maas-api-tls\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:25:52.522344 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:52.519290 4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/441dfaf9-d63b-4882-a70b-f1769a32ca9d-metrics-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:25:52.522344 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:52.519307 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkk5h\" (UniqueName: \"kubernetes.io/projected/441dfaf9-d63b-4882-a70b-f1769a32ca9d-kube-api-access-wkk5h\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:25:53.033760 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:53.033722 4931 generic.go:356] "Generic (PLEG): container finished" podID="441dfaf9-d63b-4882-a70b-f1769a32ca9d" containerID="312305c9682e71a54ccd183a2455f6906ac79748c0b0645f794192b271291920" exitCode=0 Oct 30 13:25:53.034172 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:53.033764 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv" Oct 30 13:25:53.034172 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:53.033791 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv" event={"ID":"441dfaf9-d63b-4882-a70b-f1769a32ca9d","Type":"ContainerDied","Data":"312305c9682e71a54ccd183a2455f6906ac79748c0b0645f794192b271291920"} Oct 30 13:25:53.034172 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:53.033845 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv" event={"ID":"441dfaf9-d63b-4882-a70b-f1769a32ca9d","Type":"ContainerDied","Data":"9b0729780b8deeb8e16b5581cc4a740586423581bcfa5ec800632e25b32f0112"} Oct 30 13:25:53.034172 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:53.033872 4931 scope.go:122] "RemoveContainer" containerID="312305c9682e71a54ccd183a2455f6906ac79748c0b0645f794192b271291920" Oct 30 13:25:53.044231 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:53.044208 4931 scope.go:122] "RemoveContainer" containerID="312305c9682e71a54ccd183a2455f6906ac79748c0b0645f794192b271291920" Oct 30 13:25:53.044714 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:25:53.044558 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"312305c9682e71a54ccd183a2455f6906ac79748c0b0645f794192b271291920\": container with ID starting with 312305c9682e71a54ccd183a2455f6906ac79748c0b0645f794192b271291920 not found: ID does not exist" containerID="312305c9682e71a54ccd183a2455f6906ac79748c0b0645f794192b271291920" Oct 30 13:25:53.044714 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:53.044597 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"312305c9682e71a54ccd183a2455f6906ac79748c0b0645f794192b271291920"} err="failed to get container status \"312305c9682e71a54ccd183a2455f6906ac79748c0b0645f794192b271291920\": rpc error: code = NotFound desc = could not find container \"312305c9682e71a54ccd183a2455f6906ac79748c0b0645f794192b271291920\": container with ID starting with 312305c9682e71a54ccd183a2455f6906ac79748c0b0645f794192b271291920 not found: ID does not exist" Oct 30 13:25:53.106595 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:53.106568 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv"] Oct 30 13:25:53.119123 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:53.119101 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-619b9b-6ccbfc79d7-m7hgv"] Oct 30 13:25:53.514969 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:53.514926 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="441dfaf9-d63b-4882-a70b-f1769a32ca9d" path="/var/lib/kubelet/pods/441dfaf9-d63b-4882-a70b-f1769a32ca9d/volumes" Oct 30 13:25:58.884025 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:58.883952 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60a9a6-jlm9v"] Oct 30 13:25:58.885690 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:58.885661 4931 cpu_manager.go:414] "containerMap: removing container" podUID="9ba7ef30-326a-48dd-b501-940c52c0c380" containerName="revoke-keys" Oct 30 13:25:58.885854 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:58.885841 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ba7ef30-326a-48dd-b501-940c52c0c380" containerName="revoke-keys" Oct 30 13:25:58.885987 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:58.885977 4931 cpu_manager.go:414] "containerMap: removing container" podUID="441dfaf9-d63b-4882-a70b-f1769a32ca9d" containerName="maas-api" Oct 30 13:25:58.886059 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:58.886050 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="441dfaf9-d63b-4882-a70b-f1769a32ca9d" containerName="maas-api" Oct 30 13:25:58.886213 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:58.886205 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ba7ef30-326a-48dd-b501-940c52c0c380" containerName="revoke-keys" Oct 30 13:25:58.886322 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:58.886293 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="441dfaf9-d63b-4882-a70b-f1769a32ca9d" containerName="maas-api" Oct 30 13:25:58.889731 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:58.889714 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60a9a6-jlm9v" Oct 30 13:25:58.906310 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:58.906282 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60a9a6-jlm9v"] Oct 30 13:25:58.915457 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:58.915430 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:25:58.917436 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:58.917414 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:25:59.011862 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:59.011827 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzxv8\" (UniqueName: \"kubernetes.io/projected/59c64f40-68b0-46e1-b906-39d6895a640b-kube-api-access-zzxv8\") pod \"maas-api-revoke-keys-e2e-inf-a-60a9a6-jlm9v\" (UID: \"59c64f40-68b0-46e1-b906-39d6895a640b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60a9a6-jlm9v" Oct 30 13:25:59.012024 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:59.011892 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59c64f40-68b0-46e1-b906-39d6895a640b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-60a9a6-jlm9v\" (UID: \"59c64f40-68b0-46e1-b906-39d6895a640b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60a9a6-jlm9v" Oct 30 13:25:59.012127 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:59.012098 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59c64f40-68b0-46e1-b906-39d6895a640b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-60a9a6-jlm9v\" (UID: \"59c64f40-68b0-46e1-b906-39d6895a640b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60a9a6-jlm9v" Oct 30 13:25:59.113918 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:59.113871 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzxv8\" (UniqueName: \"kubernetes.io/projected/59c64f40-68b0-46e1-b906-39d6895a640b-kube-api-access-zzxv8\") pod \"maas-api-revoke-keys-e2e-inf-a-60a9a6-jlm9v\" (UID: \"59c64f40-68b0-46e1-b906-39d6895a640b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60a9a6-jlm9v" Oct 30 13:25:59.114111 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:59.113987 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59c64f40-68b0-46e1-b906-39d6895a640b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-60a9a6-jlm9v\" (UID: \"59c64f40-68b0-46e1-b906-39d6895a640b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60a9a6-jlm9v" Oct 30 13:25:59.114197 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:59.114119 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59c64f40-68b0-46e1-b906-39d6895a640b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-60a9a6-jlm9v\" (UID: \"59c64f40-68b0-46e1-b906-39d6895a640b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60a9a6-jlm9v" Oct 30 13:25:59.114633 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:59.114610 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59c64f40-68b0-46e1-b906-39d6895a640b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-60a9a6-jlm9v\" (UID: \"59c64f40-68b0-46e1-b906-39d6895a640b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60a9a6-jlm9v" Oct 30 13:25:59.116467 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:59.116447 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59c64f40-68b0-46e1-b906-39d6895a640b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-60a9a6-jlm9v\" (UID: \"59c64f40-68b0-46e1-b906-39d6895a640b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60a9a6-jlm9v" Oct 30 13:25:59.158550 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:59.158471 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzxv8\" (UniqueName: \"kubernetes.io/projected/59c64f40-68b0-46e1-b906-39d6895a640b-kube-api-access-zzxv8\") pod \"maas-api-revoke-keys-e2e-inf-a-60a9a6-jlm9v\" (UID: \"59c64f40-68b0-46e1-b906-39d6895a640b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60a9a6-jlm9v" Oct 30 13:25:59.172679 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:59.172630 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv"] Oct 30 13:25:59.173192 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:59.173143 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" podUID="78d440f8-6cfb-4f8e-905e-228d07bdcec9" containerName="istio-proxy" containerID="cri-o://d7c46df3328f7f5de418baaa55e2791ca3b2b0978746f0bc16e00a3130657e70" gracePeriod=30 Oct 30 13:25:59.203629 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:59.203601 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60a9a6-jlm9v" Oct 30 13:25:59.515877 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:59.515847 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60a9a6-jlm9v"] Oct 30 13:25:59.516171 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:25:59.516150 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59c64f40_68b0_46e1_b906_39d6895a640b.slice/crio-bd1da4bf819e4c5b836d61008b22bcb4eb1f64f8419ebc40679c8c8e768cf5b9 WatchSource:0}: Error finding container bd1da4bf819e4c5b836d61008b22bcb4eb1f64f8419ebc40679c8c8e768cf5b9: Status 404 returned error can't find the container with id bd1da4bf819e4c5b836d61008b22bcb4eb1f64f8419ebc40679c8c8e768cf5b9 Oct 30 13:25:59.576003 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:59.575974 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs"] Oct 30 13:25:59.576181 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:25:59.576155 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" podUID="655adb9b-cbfb-476b-9c20-600e99055bd0" containerName="istio-proxy" containerID="cri-o://226d7284288e4128bfe7b8d3647878631b7685596a9b06332a0566c4bf22783b" gracePeriod=30 Oct 30 13:26:00.099394 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:00.099301 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60a9a6-jlm9v" event={"ID":"59c64f40-68b0-46e1-b906-39d6895a640b","Type":"ContainerStarted","Data":"1acd0a1cdf72e17c2ec19236ae0fbbe463209ff423b810abe1cf19569a582e11"} Oct 30 13:26:00.099394 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:00.099361 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60a9a6-jlm9v" event={"ID":"59c64f40-68b0-46e1-b906-39d6895a640b","Type":"ContainerStarted","Data":"bd1da4bf819e4c5b836d61008b22bcb4eb1f64f8419ebc40679c8c8e768cf5b9"} Oct 30 13:26:00.144202 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:00.144146 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60a9a6-jlm9v" podStartSLOduration=2.144131497 podStartE2EDuration="2.144131497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:00.142481586 +0000 UTC m=+2458.745980035" watchObservedRunningTime="2026-10-30 13:26:00.144131497 +0000 UTC m=+2458.747629943" Oct 30 13:26:00.959985 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:00.959947 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8"] Oct 30 13:26:00.960248 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:00.960213 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" podUID="212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf" containerName="istio-proxy" containerID="cri-o://8c2c976c62c4e746a750e298b26681d455e45b77d7ff01162f8305916226ee33" gracePeriod=30 Oct 30 13:26:01.108568 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:01.108525 4931 generic.go:356] "Generic (PLEG): container finished" podID="59c64f40-68b0-46e1-b906-39d6895a640b" containerID="1acd0a1cdf72e17c2ec19236ae0fbbe463209ff423b810abe1cf19569a582e11" exitCode=0 Oct 30 13:26:01.108970 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:01.108656 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60a9a6-jlm9v" event={"ID":"59c64f40-68b0-46e1-b906-39d6895a640b","Type":"ContainerDied","Data":"1acd0a1cdf72e17c2ec19236ae0fbbe463209ff423b810abe1cf19569a582e11"} Oct 30 13:26:02.317942 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:02.317910 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60a9a6-jlm9v" Oct 30 13:26:02.372886 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:02.372792 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59c64f40-68b0-46e1-b906-39d6895a640b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59c64f40-68b0-46e1-b906-39d6895a640b-maas-api-openshift-ca\") pod \"59c64f40-68b0-46e1-b906-39d6895a640b\" (UID: \"59c64f40-68b0-46e1-b906-39d6895a640b\") " Oct 30 13:26:02.372886 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:02.372861 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59c64f40-68b0-46e1-b906-39d6895a640b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59c64f40-68b0-46e1-b906-39d6895a640b-maas-api-xks-ca\") pod \"59c64f40-68b0-46e1-b906-39d6895a640b\" (UID: \"59c64f40-68b0-46e1-b906-39d6895a640b\") " Oct 30 13:26:02.373134 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:02.373022 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59c64f40-68b0-46e1-b906-39d6895a640b-kube-api-access-zzxv8\" (UniqueName: \"kubernetes.io/projected/59c64f40-68b0-46e1-b906-39d6895a640b-kube-api-access-zzxv8\") pod \"59c64f40-68b0-46e1-b906-39d6895a640b\" (UID: \"59c64f40-68b0-46e1-b906-39d6895a640b\") " Oct 30 13:26:02.373223 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:02.373192 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59c64f40-68b0-46e1-b906-39d6895a640b-maas-api-openshift-ca" pod "59c64f40-68b0-46e1-b906-39d6895a640b" (UID: "59c64f40-68b0-46e1-b906-39d6895a640b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:26:02.373645 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:02.373617 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59c64f40-68b0-46e1-b906-39d6895a640b-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:02.375191 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:02.375167 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59c64f40-68b0-46e1-b906-39d6895a640b-maas-api-xks-ca" pod "59c64f40-68b0-46e1-b906-39d6895a640b" (UID: "59c64f40-68b0-46e1-b906-39d6895a640b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:26:02.375771 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:02.375745 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59c64f40-68b0-46e1-b906-39d6895a640b-kube-api-access-zzxv8" pod "59c64f40-68b0-46e1-b906-39d6895a640b" (UID: "59c64f40-68b0-46e1-b906-39d6895a640b"). InnerVolumeSpecName "kube-api-access-zzxv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:02.475492 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:02.475449 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzxv8\" (UniqueName: \"kubernetes.io/projected/59c64f40-68b0-46e1-b906-39d6895a640b-kube-api-access-zzxv8\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:02.475492 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:02.475494 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59c64f40-68b0-46e1-b906-39d6895a640b-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:02.849799 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:02.849754 4931 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:26:02.849992 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:02.849854 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" podUID="655adb9b-cbfb-476b-9c20-600e99055bd0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:26:03.127735 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:03.127651 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60a9a6-jlm9v" Oct 30 13:26:03.127735 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:03.127690 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60a9a6-jlm9v" event={"ID":"59c64f40-68b0-46e1-b906-39d6895a640b","Type":"ContainerDied","Data":"bd1da4bf819e4c5b836d61008b22bcb4eb1f64f8419ebc40679c8c8e768cf5b9"} Oct 30 13:26:03.127924 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:03.127750 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd1da4bf819e4c5b836d61008b22bcb4eb1f64f8419ebc40679c8c8e768cf5b9" Oct 30 13:26:03.209168 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:03.209129 4931 scope.go:122] "RemoveContainer" containerID="9226e5e4f51d6432d95b7ae9cfb12ec5cde6f93143829645f99958feab6b8276" Oct 30 13:26:04.882191 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:04.882165 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" Oct 30 13:26:04.903463 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:04.903437 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" Oct 30 13:26:05.016504 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.016471 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/655adb9b-cbfb-476b-9c20-600e99055bd0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/655adb9b-cbfb-476b-9c20-600e99055bd0-workload-certs\") pod \"655adb9b-cbfb-476b-9c20-600e99055bd0\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " Oct 30 13:26:05.016699 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.016514 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/655adb9b-cbfb-476b-9c20-600e99055bd0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/655adb9b-cbfb-476b-9c20-600e99055bd0-istiod-ca-cert\") pod \"655adb9b-cbfb-476b-9c20-600e99055bd0\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " Oct 30 13:26:05.016699 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.016544 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78d440f8-6cfb-4f8e-905e-228d07bdcec9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78d440f8-6cfb-4f8e-905e-228d07bdcec9-istiod-ca-cert\") pod \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " Oct 30 13:26:05.016699 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.016582 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78d440f8-6cfb-4f8e-905e-228d07bdcec9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d440f8-6cfb-4f8e-905e-228d07bdcec9-workload-socket\") pod \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " Oct 30 13:26:05.016699 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.016632 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78d440f8-6cfb-4f8e-905e-228d07bdcec9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d440f8-6cfb-4f8e-905e-228d07bdcec9-credential-socket\") pod \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " Oct 30 13:26:05.016939 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.016702 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/655adb9b-cbfb-476b-9c20-600e99055bd0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/655adb9b-cbfb-476b-9c20-600e99055bd0-istio-envoy\") pod \"655adb9b-cbfb-476b-9c20-600e99055bd0\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " Oct 30 13:26:05.016939 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.016738 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/655adb9b-cbfb-476b-9c20-600e99055bd0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/655adb9b-cbfb-476b-9c20-600e99055bd0-credential-socket\") pod \"655adb9b-cbfb-476b-9c20-600e99055bd0\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " Oct 30 13:26:05.016939 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.016761 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/655adb9b-cbfb-476b-9c20-600e99055bd0-kube-api-access-59p27\" (UniqueName: \"kubernetes.io/projected/655adb9b-cbfb-476b-9c20-600e99055bd0-kube-api-access-59p27\") pod \"655adb9b-cbfb-476b-9c20-600e99055bd0\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " Oct 30 13:26:05.016939 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.016806 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/655adb9b-cbfb-476b-9c20-600e99055bd0-workload-certs" pod "655adb9b-cbfb-476b-9c20-600e99055bd0" (UID: "655adb9b-cbfb-476b-9c20-600e99055bd0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:05.016939 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.016826 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78d440f8-6cfb-4f8e-905e-228d07bdcec9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78d440f8-6cfb-4f8e-905e-228d07bdcec9-istio-data\") pod \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " Oct 30 13:26:05.016939 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.016879 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d440f8-6cfb-4f8e-905e-228d07bdcec9-workload-socket" pod "78d440f8-6cfb-4f8e-905e-228d07bdcec9" (UID: "78d440f8-6cfb-4f8e-905e-228d07bdcec9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:05.017340 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.016942 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/655adb9b-cbfb-476b-9c20-600e99055bd0-istio-token\" (UniqueName: \"kubernetes.io/projected/655adb9b-cbfb-476b-9c20-600e99055bd0-istio-token\") pod \"655adb9b-cbfb-476b-9c20-600e99055bd0\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " Oct 30 13:26:05.017340 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.016977 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78d440f8-6cfb-4f8e-905e-228d07bdcec9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78d440f8-6cfb-4f8e-905e-228d07bdcec9-istio-podinfo\") pod \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " Oct 30 13:26:05.017340 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.017031 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/655adb9b-cbfb-476b-9c20-600e99055bd0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/655adb9b-cbfb-476b-9c20-600e99055bd0-istio-data\") pod \"655adb9b-cbfb-476b-9c20-600e99055bd0\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " Oct 30 13:26:05.017340 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.017063 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78d440f8-6cfb-4f8e-905e-228d07bdcec9-kube-api-access-6rz44\" (UniqueName: \"kubernetes.io/projected/78d440f8-6cfb-4f8e-905e-228d07bdcec9-kube-api-access-6rz44\") pod \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " Oct 30 13:26:05.017340 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.017064 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/655adb9b-cbfb-476b-9c20-600e99055bd0-istiod-ca-cert" pod "655adb9b-cbfb-476b-9c20-600e99055bd0" (UID: "655adb9b-cbfb-476b-9c20-600e99055bd0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:26:05.017340 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.017090 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78d440f8-6cfb-4f8e-905e-228d07bdcec9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78d440f8-6cfb-4f8e-905e-228d07bdcec9-workload-certs\") pod \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " Oct 30 13:26:05.017340 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.017164 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78d440f8-6cfb-4f8e-905e-228d07bdcec9-istio-token\" (UniqueName: \"kubernetes.io/projected/78d440f8-6cfb-4f8e-905e-228d07bdcec9-istio-token\") pod \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " Oct 30 13:26:05.017340 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.017245 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/655adb9b-cbfb-476b-9c20-600e99055bd0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/655adb9b-cbfb-476b-9c20-600e99055bd0-istio-podinfo\") pod \"655adb9b-cbfb-476b-9c20-600e99055bd0\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " Oct 30 13:26:05.017340 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.017303 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/655adb9b-cbfb-476b-9c20-600e99055bd0-istio-data" pod "655adb9b-cbfb-476b-9c20-600e99055bd0" (UID: "655adb9b-cbfb-476b-9c20-600e99055bd0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:05.017340 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.017313 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78d440f8-6cfb-4f8e-905e-228d07bdcec9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78d440f8-6cfb-4f8e-905e-228d07bdcec9-istio-envoy\") pod \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\" (UID: \"78d440f8-6cfb-4f8e-905e-228d07bdcec9\") " Oct 30 13:26:05.017872 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.017354 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/655adb9b-cbfb-476b-9c20-600e99055bd0-credential-socket" pod "655adb9b-cbfb-476b-9c20-600e99055bd0" (UID: "655adb9b-cbfb-476b-9c20-600e99055bd0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:05.017872 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.017370 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78d440f8-6cfb-4f8e-905e-228d07bdcec9-istiod-ca-cert" pod "78d440f8-6cfb-4f8e-905e-228d07bdcec9" (UID: "78d440f8-6cfb-4f8e-905e-228d07bdcec9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:26:05.017872 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.017385 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/655adb9b-cbfb-476b-9c20-600e99055bd0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/655adb9b-cbfb-476b-9c20-600e99055bd0-workload-socket\") pod \"655adb9b-cbfb-476b-9c20-600e99055bd0\" (UID: \"655adb9b-cbfb-476b-9c20-600e99055bd0\") " Oct 30 13:26:05.017872 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.017476 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d440f8-6cfb-4f8e-905e-228d07bdcec9-credential-socket" pod "78d440f8-6cfb-4f8e-905e-228d07bdcec9" (UID: "78d440f8-6cfb-4f8e-905e-228d07bdcec9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:05.018098 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.017946 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d440f8-6cfb-4f8e-905e-228d07bdcec9-workload-certs" pod "78d440f8-6cfb-4f8e-905e-228d07bdcec9" (UID: "78d440f8-6cfb-4f8e-905e-228d07bdcec9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:05.018161 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.018145 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/655adb9b-cbfb-476b-9c20-600e99055bd0-workload-socket" pod "655adb9b-cbfb-476b-9c20-600e99055bd0" (UID: "655adb9b-cbfb-476b-9c20-600e99055bd0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:05.018234 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.018197 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/655adb9b-cbfb-476b-9c20-600e99055bd0-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:05.018234 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.018216 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/655adb9b-cbfb-476b-9c20-600e99055bd0-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:05.018234 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.018231 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78d440f8-6cfb-4f8e-905e-228d07bdcec9-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:05.018420 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.018244 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d440f8-6cfb-4f8e-905e-228d07bdcec9-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:05.018420 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.018258 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d440f8-6cfb-4f8e-905e-228d07bdcec9-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:05.018420 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.018289 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/655adb9b-cbfb-476b-9c20-600e99055bd0-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:05.018420 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.018303 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/655adb9b-cbfb-476b-9c20-600e99055bd0-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:05.018420 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.018317 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78d440f8-6cfb-4f8e-905e-228d07bdcec9-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:05.018930 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.018906 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d440f8-6cfb-4f8e-905e-228d07bdcec9-istio-data" pod "78d440f8-6cfb-4f8e-905e-228d07bdcec9" (UID: "78d440f8-6cfb-4f8e-905e-228d07bdcec9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:05.020011 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.019989 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/655adb9b-cbfb-476b-9c20-600e99055bd0-istio-token" pod "655adb9b-cbfb-476b-9c20-600e99055bd0" (UID: "655adb9b-cbfb-476b-9c20-600e99055bd0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:05.020545 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.020508 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d440f8-6cfb-4f8e-905e-228d07bdcec9-istio-envoy" pod "78d440f8-6cfb-4f8e-905e-228d07bdcec9" (UID: "78d440f8-6cfb-4f8e-905e-228d07bdcec9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:05.020686 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.020614 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78d440f8-6cfb-4f8e-905e-228d07bdcec9-istio-podinfo" pod "78d440f8-6cfb-4f8e-905e-228d07bdcec9" (UID: "78d440f8-6cfb-4f8e-905e-228d07bdcec9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:26:05.020686 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.020680 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d440f8-6cfb-4f8e-905e-228d07bdcec9-istio-token" pod "78d440f8-6cfb-4f8e-905e-228d07bdcec9" (UID: "78d440f8-6cfb-4f8e-905e-228d07bdcec9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:05.021252 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.021226 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d440f8-6cfb-4f8e-905e-228d07bdcec9-kube-api-access-6rz44" pod "78d440f8-6cfb-4f8e-905e-228d07bdcec9" (UID: "78d440f8-6cfb-4f8e-905e-228d07bdcec9"). InnerVolumeSpecName "kube-api-access-6rz44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:05.021875 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.021856 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/655adb9b-cbfb-476b-9c20-600e99055bd0-istio-podinfo" pod "655adb9b-cbfb-476b-9c20-600e99055bd0" (UID: "655adb9b-cbfb-476b-9c20-600e99055bd0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:26:05.022856 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.022836 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/655adb9b-cbfb-476b-9c20-600e99055bd0-istio-envoy" pod "655adb9b-cbfb-476b-9c20-600e99055bd0" (UID: "655adb9b-cbfb-476b-9c20-600e99055bd0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:05.023934 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.023905 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/655adb9b-cbfb-476b-9c20-600e99055bd0-kube-api-access-59p27" pod "655adb9b-cbfb-476b-9c20-600e99055bd0" (UID: "655adb9b-cbfb-476b-9c20-600e99055bd0"). InnerVolumeSpecName "kube-api-access-59p27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:05.119807 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.119723 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59p27\" (UniqueName: \"kubernetes.io/projected/655adb9b-cbfb-476b-9c20-600e99055bd0-kube-api-access-59p27\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:05.119807 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.119755 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78d440f8-6cfb-4f8e-905e-228d07bdcec9-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:05.119807 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.119764 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/655adb9b-cbfb-476b-9c20-600e99055bd0-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:05.119807 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.119773 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78d440f8-6cfb-4f8e-905e-228d07bdcec9-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:05.119807 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.119781 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rz44\" (UniqueName: \"kubernetes.io/projected/78d440f8-6cfb-4f8e-905e-228d07bdcec9-kube-api-access-6rz44\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:05.119807 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.119789 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78d440f8-6cfb-4f8e-905e-228d07bdcec9-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:05.119807 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.119800 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/655adb9b-cbfb-476b-9c20-600e99055bd0-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:05.119807 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.119809 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78d440f8-6cfb-4f8e-905e-228d07bdcec9-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:05.120312 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.119817 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/655adb9b-cbfb-476b-9c20-600e99055bd0-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:05.120312 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.119825 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/655adb9b-cbfb-476b-9c20-600e99055bd0-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:05.143753 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.143722 4931 generic.go:356] "Generic (PLEG): container finished" podID="655adb9b-cbfb-476b-9c20-600e99055bd0" containerID="226d7284288e4128bfe7b8d3647878631b7685596a9b06332a0566c4bf22783b" exitCode=0 Oct 30 13:26:05.143929 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.143770 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" Oct 30 13:26:05.143929 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.143800 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" event={"ID":"655adb9b-cbfb-476b-9c20-600e99055bd0","Type":"ContainerDied","Data":"226d7284288e4128bfe7b8d3647878631b7685596a9b06332a0566c4bf22783b"} Oct 30 13:26:05.143929 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.143865 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs" event={"ID":"655adb9b-cbfb-476b-9c20-600e99055bd0","Type":"ContainerDied","Data":"0deaddf130994a56d065855e6dbba80c9b9c8ee254ec4895ec3c012b98b0f5c9"} Oct 30 13:26:05.143929 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.143896 4931 scope.go:122] "RemoveContainer" containerID="226d7284288e4128bfe7b8d3647878631b7685596a9b06332a0566c4bf22783b" Oct 30 13:26:05.145481 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.145460 4931 generic.go:356] "Generic (PLEG): container finished" podID="78d440f8-6cfb-4f8e-905e-228d07bdcec9" containerID="d7c46df3328f7f5de418baaa55e2791ca3b2b0978746f0bc16e00a3130657e70" exitCode=0 Oct 30 13:26:05.145590 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.145509 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" Oct 30 13:26:05.145590 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.145568 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" event={"ID":"78d440f8-6cfb-4f8e-905e-228d07bdcec9","Type":"ContainerDied","Data":"d7c46df3328f7f5de418baaa55e2791ca3b2b0978746f0bc16e00a3130657e70"} Oct 30 13:26:05.145695 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.145618 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv" event={"ID":"78d440f8-6cfb-4f8e-905e-228d07bdcec9","Type":"ContainerDied","Data":"1b4504716a5489e3781f0be0a063fcf9f683ad26e91e19cd4181eff575521e63"} Oct 30 13:26:05.155074 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.155006 4931 scope.go:122] "RemoveContainer" containerID="226d7284288e4128bfe7b8d3647878631b7685596a9b06332a0566c4bf22783b" Oct 30 13:26:05.155438 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:26:05.155418 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"226d7284288e4128bfe7b8d3647878631b7685596a9b06332a0566c4bf22783b\": container with ID starting with 226d7284288e4128bfe7b8d3647878631b7685596a9b06332a0566c4bf22783b not found: ID does not exist" containerID="226d7284288e4128bfe7b8d3647878631b7685596a9b06332a0566c4bf22783b" Oct 30 13:26:05.155518 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.155447 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"226d7284288e4128bfe7b8d3647878631b7685596a9b06332a0566c4bf22783b"} err="failed to get container status \"226d7284288e4128bfe7b8d3647878631b7685596a9b06332a0566c4bf22783b\": rpc error: code = NotFound desc = could not find container \"226d7284288e4128bfe7b8d3647878631b7685596a9b06332a0566c4bf22783b\": container with ID starting with 226d7284288e4128bfe7b8d3647878631b7685596a9b06332a0566c4bf22783b not found: ID does not exist" Oct 30 13:26:05.155518 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.155471 4931 scope.go:122] "RemoveContainer" containerID="d7c46df3328f7f5de418baaa55e2791ca3b2b0978746f0bc16e00a3130657e70" Oct 30 13:26:05.167491 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.167471 4931 scope.go:122] "RemoveContainer" containerID="d7c46df3328f7f5de418baaa55e2791ca3b2b0978746f0bc16e00a3130657e70" Oct 30 13:26:05.167772 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:26:05.167747 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7c46df3328f7f5de418baaa55e2791ca3b2b0978746f0bc16e00a3130657e70\": container with ID starting with d7c46df3328f7f5de418baaa55e2791ca3b2b0978746f0bc16e00a3130657e70 not found: ID does not exist" containerID="d7c46df3328f7f5de418baaa55e2791ca3b2b0978746f0bc16e00a3130657e70" Oct 30 13:26:05.167859 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.167782 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7c46df3328f7f5de418baaa55e2791ca3b2b0978746f0bc16e00a3130657e70"} err="failed to get container status \"d7c46df3328f7f5de418baaa55e2791ca3b2b0978746f0bc16e00a3130657e70\": rpc error: code = NotFound desc = could not find container \"d7c46df3328f7f5de418baaa55e2791ca3b2b0978746f0bc16e00a3130657e70\": container with ID starting with d7c46df3328f7f5de418baaa55e2791ca3b2b0978746f0bc16e00a3130657e70 not found: ID does not exist" Oct 30 13:26:05.207315 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.206256 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv"] Oct 30 13:26:05.215770 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.215742 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv"] Oct 30 13:26:05.259884 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.259840 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs"] Oct 30 13:26:05.263081 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.263058 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs"] Oct 30 13:26:05.288675 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.288643 4931 patch_prober.go:29] interesting pod/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:26:05.288807 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.288718 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" podUID="212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:26:05.514246 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.514211 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="655adb9b-cbfb-476b-9c20-600e99055bd0" path="/var/lib/kubelet/pods/655adb9b-cbfb-476b-9c20-600e99055bd0/volumes" Oct 30 13:26:05.514788 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:05.514770 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78d440f8-6cfb-4f8e-905e-228d07bdcec9" path="/var/lib/kubelet/pods/78d440f8-6cfb-4f8e-905e-228d07bdcec9/volumes" Oct 30 13:26:06.160835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:06.160798 4931 generic.go:356] "Generic (PLEG): container finished" podID="212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf" containerID="8c2c976c62c4e746a750e298b26681d455e45b77d7ff01162f8305916226ee33" exitCode=0 Oct 30 13:26:06.160835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:06.160831 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" event={"ID":"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf","Type":"ContainerDied","Data":"8c2c976c62c4e746a750e298b26681d455e45b77d7ff01162f8305916226ee33"} Oct 30 13:26:06.161410 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:06.160905 4931 scope.go:122] "RemoveContainer" containerID="f37ca073e38d4e6229125b4e00a5e9b5222822eaa461c4de7621af75389f1f98" Oct 30 13:26:06.244898 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:06.244826 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:26:06.336714 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:06.336577 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-workload-socket\") pod \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " Oct 30 13:26:06.336714 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:06.336634 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-credential-socket\") pod \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " Oct 30 13:26:06.336714 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:06.336711 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-kube-api-access-fdk6s\" (UniqueName: \"kubernetes.io/projected/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-kube-api-access-fdk6s\") pod \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " Oct 30 13:26:06.337021 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:06.336818 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-workload-certs\") pod \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " Oct 30 13:26:06.337021 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:06.336835 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-workload-socket" pod "212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf" (UID: "212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:06.337021 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:06.336921 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-istio-token\" (UniqueName: \"kubernetes.io/projected/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-istio-token\") pod \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " Oct 30 13:26:06.337021 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:06.336964 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-istio-envoy\") pod \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " Oct 30 13:26:06.337240 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:06.337032 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-istio-podinfo\") pod \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " Oct 30 13:26:06.337240 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:06.337082 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-istio-data\") pod \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " Oct 30 13:26:06.337240 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:06.337114 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-istiod-ca-cert\") pod \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\" (UID: \"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf\") " Oct 30 13:26:06.337240 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:06.337128 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-workload-certs" pod "212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf" (UID: "212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:06.337643 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:06.337255 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-credential-socket" pod "212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf" (UID: "212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:06.337643 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:06.337530 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-istio-data" pod "212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf" (UID: "212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:06.337751 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:06.337691 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-istiod-ca-cert" pod "212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf" (UID: "212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:26:06.337808 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:06.337763 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:06.337808 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:06.337784 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:06.337808 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:06.337797 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:06.337978 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:06.337814 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:06.339489 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:06.339462 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-kube-api-access-fdk6s" pod "212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf" (UID: "212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf"). InnerVolumeSpecName "kube-api-access-fdk6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:06.339661 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:06.339635 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-istio-podinfo" pod "212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf" (UID: "212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:26:06.340016 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:06.339995 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-istio-envoy" pod "212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf" (UID: "212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:06.340356 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:06.340339 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-istio-token" pod "212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf" (UID: "212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:06.438870 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:06.438836 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdk6s\" (UniqueName: \"kubernetes.io/projected/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-kube-api-access-fdk6s\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:06.438870 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:06.438865 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:06.438870 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:06.438875 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:06.439124 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:06.438884 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:06.439124 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:06.438892 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:07.170049 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:07.170019 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" Oct 30 13:26:07.170529 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:07.170055 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8" event={"ID":"212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf","Type":"ContainerDied","Data":"8ba002a8ac598fb1491ed425d29cf32172f4a7721a5386ae6d067de49da4edf1"} Oct 30 13:26:07.170529 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:07.170123 4931 scope.go:122] "RemoveContainer" containerID="8c2c976c62c4e746a750e298b26681d455e45b77d7ff01162f8305916226ee33" Oct 30 13:26:07.263865 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:07.263826 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8"] Oct 30 13:26:07.275946 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:07.275915 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8"] Oct 30 13:26:07.353905 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:26:07.353863 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:26:07.513299 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:07.513251 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf" path="/var/lib/kubelet/pods/212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf/volumes" Oct 30 13:26:31.660713 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.660673 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs"] Oct 30 13:26:31.661548 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.661524 4931 cpu_manager.go:414] "containerMap: removing container" podUID="212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf" containerName="istio-proxy" Oct 30 13:26:31.661644 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.661550 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf" containerName="istio-proxy" Oct 30 13:26:31.661644 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.661602 4931 cpu_manager.go:414] "containerMap: removing container" podUID="655adb9b-cbfb-476b-9c20-600e99055bd0" containerName="istio-proxy" Oct 30 13:26:31.661644 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.661611 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="655adb9b-cbfb-476b-9c20-600e99055bd0" containerName="istio-proxy" Oct 30 13:26:31.661644 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.661627 4931 cpu_manager.go:414] "containerMap: removing container" podUID="78d440f8-6cfb-4f8e-905e-228d07bdcec9" containerName="istio-proxy" Oct 30 13:26:31.661644 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.661635 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78d440f8-6cfb-4f8e-905e-228d07bdcec9" containerName="istio-proxy" Oct 30 13:26:31.661831 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.661667 4931 cpu_manager.go:414] "containerMap: removing container" podUID="59c64f40-68b0-46e1-b906-39d6895a640b" containerName="revoke-keys" Oct 30 13:26:31.661831 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.661675 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59c64f40-68b0-46e1-b906-39d6895a640b" containerName="revoke-keys" Oct 30 13:26:31.661918 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.661835 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf" containerName="istio-proxy" Oct 30 13:26:31.661918 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.661859 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="655adb9b-cbfb-476b-9c20-600e99055bd0" containerName="istio-proxy" Oct 30 13:26:31.661918 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.661872 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="78d440f8-6cfb-4f8e-905e-228d07bdcec9" containerName="istio-proxy" Oct 30 13:26:31.661918 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.661893 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="59c64f40-68b0-46e1-b906-39d6895a640b" containerName="revoke-keys" Oct 30 13:26:31.661918 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.661901 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf" containerName="istio-proxy" Oct 30 13:26:31.666050 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.666018 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs" Oct 30 13:26:31.687449 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.687416 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs"] Oct 30 13:26:31.694777 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.694753 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-7a71ca\"/\"kube-root-ca.crt\"" Oct 30 13:26:31.694777 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.694754 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-7a71ca\"/\"openshift-service-ca.crt\"" Oct 30 13:26:31.701127 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.701101 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-7a71ca\"/\"default-dockercfg-snzmh\"" Oct 30 13:26:31.701261 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.701146 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-7a71ca\"/\"auth-test-model-7a71ca-kserve-self-signed-certs\"" Oct 30 13:26:31.786216 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.786178 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6r4v\" (UniqueName: \"kubernetes.io/projected/8ca147ee-5965-483a-a4a2-7383326e09b0-kube-api-access-t6r4v\") pod \"auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs\" (UID: \"8ca147ee-5965-483a-a4a2-7383326e09b0\") " pod="ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs" Oct 30 13:26:31.786216 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.786215 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ca147ee-5965-483a-a4a2-7383326e09b0-dshm\") pod \"auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs\" (UID: \"8ca147ee-5965-483a-a4a2-7383326e09b0\") " pod="ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs" Oct 30 13:26:31.786522 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.786459 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ca147ee-5965-483a-a4a2-7383326e09b0-home\") pod \"auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs\" (UID: \"8ca147ee-5965-483a-a4a2-7383326e09b0\") " pod="ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs" Oct 30 13:26:31.786522 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.786505 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ca147ee-5965-483a-a4a2-7383326e09b0-tls-certs\") pod \"auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs\" (UID: \"8ca147ee-5965-483a-a4a2-7383326e09b0\") " pod="ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs" Oct 30 13:26:31.786651 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.786570 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ca147ee-5965-483a-a4a2-7383326e09b0-model-cache\") pod \"auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs\" (UID: \"8ca147ee-5965-483a-a4a2-7383326e09b0\") " pod="ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs" Oct 30 13:26:31.786651 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.786631 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ca147ee-5965-483a-a4a2-7383326e09b0-tmp-dir\") pod \"auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs\" (UID: \"8ca147ee-5965-483a-a4a2-7383326e09b0\") " pod="ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs" Oct 30 13:26:31.888072 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.888025 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6r4v\" (UniqueName: \"kubernetes.io/projected/8ca147ee-5965-483a-a4a2-7383326e09b0-kube-api-access-t6r4v\") pod \"auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs\" (UID: \"8ca147ee-5965-483a-a4a2-7383326e09b0\") " pod="ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs" Oct 30 13:26:31.888072 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.888075 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ca147ee-5965-483a-a4a2-7383326e09b0-dshm\") pod \"auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs\" (UID: \"8ca147ee-5965-483a-a4a2-7383326e09b0\") " pod="ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs" Oct 30 13:26:31.888408 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.888220 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ca147ee-5965-483a-a4a2-7383326e09b0-home\") pod \"auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs\" (UID: \"8ca147ee-5965-483a-a4a2-7383326e09b0\") " pod="ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs" Oct 30 13:26:31.888408 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.888262 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ca147ee-5965-483a-a4a2-7383326e09b0-tls-certs\") pod \"auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs\" (UID: \"8ca147ee-5965-483a-a4a2-7383326e09b0\") " pod="ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs" Oct 30 13:26:31.888408 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.888344 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ca147ee-5965-483a-a4a2-7383326e09b0-model-cache\") pod \"auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs\" (UID: \"8ca147ee-5965-483a-a4a2-7383326e09b0\") " pod="ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs" Oct 30 13:26:31.888408 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.888386 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ca147ee-5965-483a-a4a2-7383326e09b0-tmp-dir\") pod \"auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs\" (UID: \"8ca147ee-5965-483a-a4a2-7383326e09b0\") " pod="ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs" Oct 30 13:26:31.888831 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.888799 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ca147ee-5965-483a-a4a2-7383326e09b0-tmp-dir\") pod \"auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs\" (UID: \"8ca147ee-5965-483a-a4a2-7383326e09b0\") " pod="ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs" Oct 30 13:26:31.888972 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.888858 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ca147ee-5965-483a-a4a2-7383326e09b0-model-cache\") pod \"auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs\" (UID: \"8ca147ee-5965-483a-a4a2-7383326e09b0\") " pod="ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs" Oct 30 13:26:31.889039 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.888999 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ca147ee-5965-483a-a4a2-7383326e09b0-home\") pod \"auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs\" (UID: \"8ca147ee-5965-483a-a4a2-7383326e09b0\") " pod="ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs" Oct 30 13:26:31.891180 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.891149 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ca147ee-5965-483a-a4a2-7383326e09b0-dshm\") pod \"auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs\" (UID: \"8ca147ee-5965-483a-a4a2-7383326e09b0\") " pod="ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs" Oct 30 13:26:31.891462 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.891440 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ca147ee-5965-483a-a4a2-7383326e09b0-tls-certs\") pod \"auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs\" (UID: \"8ca147ee-5965-483a-a4a2-7383326e09b0\") " pod="ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs" Oct 30 13:26:31.929157 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.929075 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6r4v\" (UniqueName: \"kubernetes.io/projected/8ca147ee-5965-483a-a4a2-7383326e09b0-kube-api-access-t6r4v\") pod \"auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs\" (UID: \"8ca147ee-5965-483a-a4a2-7383326e09b0\") " pod="ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs" Oct 30 13:26:31.990289 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:31.982533 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs" Oct 30 13:26:32.323989 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:32.323953 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs"] Oct 30 13:26:32.324811 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:26:32.324781 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ca147ee_5965_483a_a4a2_7383326e09b0.slice/crio-12a314a28894dbf5244ff95e6510ea173ee247ee61514c7e21d359e577004f7c WatchSource:0}: Error finding container 12a314a28894dbf5244ff95e6510ea173ee247ee61514c7e21d359e577004f7c: Status 404 returned error can't find the container with id 12a314a28894dbf5244ff95e6510ea173ee247ee61514c7e21d359e577004f7c Oct 30 13:26:32.356008 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:32.355969 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs" event={"ID":"8ca147ee-5965-483a-a4a2-7383326e09b0","Type":"ContainerStarted","Data":"12a314a28894dbf5244ff95e6510ea173ee247ee61514c7e21d359e577004f7c"} Oct 30 13:26:33.365376 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:33.365337 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs" event={"ID":"8ca147ee-5965-483a-a4a2-7383326e09b0","Type":"ContainerStarted","Data":"6e34d813fb649724517a3881a302b371dc1d7b84099b008e6eb7b72949a6fe6d"} Oct 30 13:26:33.417095 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:33.417035 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs" podStartSLOduration=2.41701914 podStartE2EDuration="2.41701914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:33.415708207 +0000 UTC m=+2492.019206655" watchObservedRunningTime="2026-10-30 13:26:33.41701914 +0000 UTC m=+2492.020517585" Oct 30 13:26:33.983660 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:33.983610 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs" Oct 30 13:26:37.354042 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:26:37.354001 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:26:41.983293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:41.983224 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs" Oct 30 13:26:41.995425 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:41.995403 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs" Oct 30 13:26:42.443387 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:42.443360 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs" Oct 30 13:26:45.905577 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:45.905533 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9"] Oct 30 13:26:45.906099 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:45.905787 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9" podUID="1a6eeedf-7968-49d9-a1f0-d8291c719f8c" containerName="maas-api" containerID="cri-o://684ebce8f863e5de3c8d24e18b8758096225ce0dc50fafe674e89fff885638f7" gracePeriod=30 Oct 30 13:26:46.078796 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:46.078752 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9" podUID="1a6eeedf-7968-49d9-a1f0-d8291c719f8c" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.156:8443/health\": dial tcp 10.132.0.156:8443: connect: connection refused" Oct 30 13:26:46.318606 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:46.318578 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9" Oct 30 13:26:46.455386 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:46.455042 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a6eeedf-7968-49d9-a1f0-d8291c719f8c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a6eeedf-7968-49d9-a1f0-d8291c719f8c-metrics-certs\") pod \"1a6eeedf-7968-49d9-a1f0-d8291c719f8c\" (UID: \"1a6eeedf-7968-49d9-a1f0-d8291c719f8c\") " Oct 30 13:26:46.455386 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:46.455186 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a6eeedf-7968-49d9-a1f0-d8291c719f8c-kube-api-access-7jgbd\" (UniqueName: \"kubernetes.io/projected/1a6eeedf-7968-49d9-a1f0-d8291c719f8c-kube-api-access-7jgbd\") pod \"1a6eeedf-7968-49d9-a1f0-d8291c719f8c\" (UID: \"1a6eeedf-7968-49d9-a1f0-d8291c719f8c\") " Oct 30 13:26:46.455386 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:46.455230 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a6eeedf-7968-49d9-a1f0-d8291c719f8c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a6eeedf-7968-49d9-a1f0-d8291c719f8c-maas-api-tls\") pod \"1a6eeedf-7968-49d9-a1f0-d8291c719f8c\" (UID: \"1a6eeedf-7968-49d9-a1f0-d8291c719f8c\") " Oct 30 13:26:46.455707 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:46.455417 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a6eeedf-7968-49d9-a1f0-d8291c719f8c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a6eeedf-7968-49d9-a1f0-d8291c719f8c-openshift-service-ca\") pod \"1a6eeedf-7968-49d9-a1f0-d8291c719f8c\" (UID: \"1a6eeedf-7968-49d9-a1f0-d8291c719f8c\") " Oct 30 13:26:46.460134 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:46.459854 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a6eeedf-7968-49d9-a1f0-d8291c719f8c-kube-api-access-7jgbd" pod "1a6eeedf-7968-49d9-a1f0-d8291c719f8c" (UID: "1a6eeedf-7968-49d9-a1f0-d8291c719f8c"). InnerVolumeSpecName "kube-api-access-7jgbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:46.462585 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:46.461186 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jgbd\" (UniqueName: \"kubernetes.io/projected/1a6eeedf-7968-49d9-a1f0-d8291c719f8c-kube-api-access-7jgbd\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:46.462585 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:46.461561 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a6eeedf-7968-49d9-a1f0-d8291c719f8c-openshift-service-ca" pod "1a6eeedf-7968-49d9-a1f0-d8291c719f8c" (UID: "1a6eeedf-7968-49d9-a1f0-d8291c719f8c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:26:46.463870 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:46.463819 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a6eeedf-7968-49d9-a1f0-d8291c719f8c-metrics-certs" pod "1a6eeedf-7968-49d9-a1f0-d8291c719f8c" (UID: "1a6eeedf-7968-49d9-a1f0-d8291c719f8c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:26:46.467200 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:46.467089 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a6eeedf-7968-49d9-a1f0-d8291c719f8c-maas-api-tls" pod "1a6eeedf-7968-49d9-a1f0-d8291c719f8c" (UID: "1a6eeedf-7968-49d9-a1f0-d8291c719f8c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:26:46.468988 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:46.468966 4931 generic.go:356] "Generic (PLEG): container finished" podID="1a6eeedf-7968-49d9-a1f0-d8291c719f8c" containerID="684ebce8f863e5de3c8d24e18b8758096225ce0dc50fafe674e89fff885638f7" exitCode=0 Oct 30 13:26:46.469094 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:46.469013 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9" event={"ID":"1a6eeedf-7968-49d9-a1f0-d8291c719f8c","Type":"ContainerDied","Data":"684ebce8f863e5de3c8d24e18b8758096225ce0dc50fafe674e89fff885638f7"} Oct 30 13:26:46.469094 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:46.469030 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9" Oct 30 13:26:46.469210 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:46.469103 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9" event={"ID":"1a6eeedf-7968-49d9-a1f0-d8291c719f8c","Type":"ContainerDied","Data":"ce78611cbc405f5cf2b3ec69aaebc0df31c9097e9c5f8a1ef83a3fba2edc08bf"} Oct 30 13:26:46.469210 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:46.469137 4931 scope.go:122] "RemoveContainer" containerID="684ebce8f863e5de3c8d24e18b8758096225ce0dc50fafe674e89fff885638f7" Oct 30 13:26:46.502085 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:46.502064 4931 scope.go:122] "RemoveContainer" containerID="684ebce8f863e5de3c8d24e18b8758096225ce0dc50fafe674e89fff885638f7" Oct 30 13:26:46.502458 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:26:46.502430 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"684ebce8f863e5de3c8d24e18b8758096225ce0dc50fafe674e89fff885638f7\": container with ID starting with 684ebce8f863e5de3c8d24e18b8758096225ce0dc50fafe674e89fff885638f7 not found: ID does not exist" containerID="684ebce8f863e5de3c8d24e18b8758096225ce0dc50fafe674e89fff885638f7" Oct 30 13:26:46.502546 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:46.502467 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"684ebce8f863e5de3c8d24e18b8758096225ce0dc50fafe674e89fff885638f7"} err="failed to get container status \"684ebce8f863e5de3c8d24e18b8758096225ce0dc50fafe674e89fff885638f7\": rpc error: code = NotFound desc = could not find container \"684ebce8f863e5de3c8d24e18b8758096225ce0dc50fafe674e89fff885638f7\": container with ID starting with 684ebce8f863e5de3c8d24e18b8758096225ce0dc50fafe674e89fff885638f7 not found: ID does not exist" Oct 30 13:26:46.542044 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:46.541965 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9"] Oct 30 13:26:46.545290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:46.545254 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-60a9a6-77f9f8b598-gr2v9"] Oct 30 13:26:46.562705 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:46.562682 4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a6eeedf-7968-49d9-a1f0-d8291c719f8c-openshift-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:46.562816 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:46.562707 4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a6eeedf-7968-49d9-a1f0-d8291c719f8c-metrics-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:46.562816 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:46.562719 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a6eeedf-7968-49d9-a1f0-d8291c719f8c-maas-api-tls\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:46.734160 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:46.734112 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60a9a6-jlm9v"] Oct 30 13:26:46.737591 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:46.737563 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60a9a6-jlm9v"] Oct 30 13:26:47.513412 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:47.513378 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a6eeedf-7968-49d9-a1f0-d8291c719f8c" path="/var/lib/kubelet/pods/1a6eeedf-7968-49d9-a1f0-d8291c719f8c/volumes" Oct 30 13:26:47.514055 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:47.514032 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59c64f40-68b0-46e1-b906-39d6895a640b" path="/var/lib/kubelet/pods/59c64f40-68b0-46e1-b906-39d6895a640b/volumes" Oct 30 13:26:48.802341 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:48.802305 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74fc974f-9fcx4"] Oct 30 13:26:48.802861 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:48.802848 4931 cpu_manager.go:414] "containerMap: removing container" podUID="1a6eeedf-7968-49d9-a1f0-d8291c719f8c" containerName="maas-api" Oct 30 13:26:48.802905 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:48.802861 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a6eeedf-7968-49d9-a1f0-d8291c719f8c" containerName="maas-api" Oct 30 13:26:48.802905 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:48.802876 4931 cpu_manager.go:414] "containerMap: removing container" podUID="212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf" containerName="istio-proxy" Oct 30 13:26:48.802905 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:48.802896 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="212428c9-b6b3-4757-9ef5-ffe8b2fa0dbf" containerName="istio-proxy" Oct 30 13:26:48.803009 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:48.802995 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a6eeedf-7968-49d9-a1f0-d8291c719f8c" containerName="maas-api" Oct 30 13:26:48.807954 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:48.807933 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fc974f-9fcx4" Oct 30 13:26:48.829135 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:48.829105 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74fc974f-9fcx4"] Oct 30 13:26:48.836986 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:48.836956 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:26:48.837155 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:48.837033 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:26:48.894690 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:48.894659 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-762fj\" (UniqueName: \"kubernetes.io/projected/5cbbc99e-cb4e-4388-a71f-fa4e87d6555e-kube-api-access-762fj\") pod \"maas-api-revoke-sub-74fc974f-9fcx4\" (UID: \"5cbbc99e-cb4e-4388-a71f-fa4e87d6555e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fc974f-9fcx4" Oct 30 13:26:48.894855 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:48.894777 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cbbc99e-cb4e-4388-a71f-fa4e87d6555e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74fc974f-9fcx4\" (UID: \"5cbbc99e-cb4e-4388-a71f-fa4e87d6555e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fc974f-9fcx4" Oct 30 13:26:48.894855 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:48.894799 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cbbc99e-cb4e-4388-a71f-fa4e87d6555e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74fc974f-9fcx4\" (UID: \"5cbbc99e-cb4e-4388-a71f-fa4e87d6555e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fc974f-9fcx4" Oct 30 13:26:48.996798 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:48.996758 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-762fj\" (UniqueName: \"kubernetes.io/projected/5cbbc99e-cb4e-4388-a71f-fa4e87d6555e-kube-api-access-762fj\") pod \"maas-api-revoke-sub-74fc974f-9fcx4\" (UID: \"5cbbc99e-cb4e-4388-a71f-fa4e87d6555e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fc974f-9fcx4" Oct 30 13:26:48.997033 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:48.996876 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cbbc99e-cb4e-4388-a71f-fa4e87d6555e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74fc974f-9fcx4\" (UID: \"5cbbc99e-cb4e-4388-a71f-fa4e87d6555e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fc974f-9fcx4" Oct 30 13:26:48.997033 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:48.996907 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cbbc99e-cb4e-4388-a71f-fa4e87d6555e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74fc974f-9fcx4\" (UID: \"5cbbc99e-cb4e-4388-a71f-fa4e87d6555e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fc974f-9fcx4" Oct 30 13:26:48.997756 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:48.997723 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cbbc99e-cb4e-4388-a71f-fa4e87d6555e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74fc974f-9fcx4\" (UID: \"5cbbc99e-cb4e-4388-a71f-fa4e87d6555e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fc974f-9fcx4" Oct 30 13:26:48.999488 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:48.999464 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cbbc99e-cb4e-4388-a71f-fa4e87d6555e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74fc974f-9fcx4\" (UID: \"5cbbc99e-cb4e-4388-a71f-fa4e87d6555e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fc974f-9fcx4" Oct 30 13:26:49.022395 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:49.022357 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-762fj\" (UniqueName: \"kubernetes.io/projected/5cbbc99e-cb4e-4388-a71f-fa4e87d6555e-kube-api-access-762fj\") pod \"maas-api-revoke-sub-74fc974f-9fcx4\" (UID: \"5cbbc99e-cb4e-4388-a71f-fa4e87d6555e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fc974f-9fcx4" Oct 30 13:26:49.123056 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:49.122971 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fc974f-9fcx4" Oct 30 13:26:49.431093 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:49.431067 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74fc974f-9fcx4"] Oct 30 13:26:49.432838 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:26:49.432805 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cbbc99e_cb4e_4388_a71f_fa4e87d6555e.slice/crio-26071c6a9de4a15d2a7276bad1cf8c8d503cc9388ce7babfd3f2888f7c2311f5 WatchSource:0}: Error finding container 26071c6a9de4a15d2a7276bad1cf8c8d503cc9388ce7babfd3f2888f7c2311f5: Status 404 returned error can't find the container with id 26071c6a9de4a15d2a7276bad1cf8c8d503cc9388ce7babfd3f2888f7c2311f5 Oct 30 13:26:49.494022 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:49.493983 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fc974f-9fcx4" event={"ID":"5cbbc99e-cb4e-4388-a71f-fa4e87d6555e","Type":"ContainerStarted","Data":"26071c6a9de4a15d2a7276bad1cf8c8d503cc9388ce7babfd3f2888f7c2311f5"} Oct 30 13:26:50.501879 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:50.501839 4931 generic.go:356] "Generic (PLEG): container finished" podID="5cbbc99e-cb4e-4388-a71f-fa4e87d6555e" containerID="3c020b3dbc9c2178cf0808d4f2a6fdfbcc9608b78ab8fc1de3ed88b5ffb2f42b" exitCode=0 Oct 30 13:26:50.502306 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:50.501935 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fc974f-9fcx4" event={"ID":"5cbbc99e-cb4e-4388-a71f-fa4e87d6555e","Type":"ContainerDied","Data":"3c020b3dbc9c2178cf0808d4f2a6fdfbcc9608b78ab8fc1de3ed88b5ffb2f42b"} Oct 30 13:26:50.785516 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:50.785483 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn"] Oct 30 13:26:50.789839 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:50.789814 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" Oct 30 13:26:50.812174 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:50.812149 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-2afe58-openshift-default-dockercfg-dhp2v\"" Oct 30 13:26:50.827276 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:50.827237 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn"] Oct 30 13:26:50.923404 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:50.923367 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qss9\" (UniqueName: \"kubernetes.io/projected/ec336656-bd10-4f0e-a598-5e5cd855b294-kube-api-access-7qss9\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" Oct 30 13:26:50.923578 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:50.923519 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec336656-bd10-4f0e-a598-5e5cd855b294-workload-certs\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" Oct 30 13:26:50.923619 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:50.923592 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec336656-bd10-4f0e-a598-5e5cd855b294-istio-envoy\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" Oct 30 13:26:50.923619 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:50.923613 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec336656-bd10-4f0e-a598-5e5cd855b294-istio-data\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" Oct 30 13:26:50.923718 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:50.923707 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec336656-bd10-4f0e-a598-5e5cd855b294-istiod-ca-cert\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" Oct 30 13:26:50.923772 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:50.923754 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec336656-bd10-4f0e-a598-5e5cd855b294-credential-socket\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" Oct 30 13:26:50.923848 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:50.923835 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec336656-bd10-4f0e-a598-5e5cd855b294-istio-token\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" Oct 30 13:26:50.923901 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:50.923879 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec336656-bd10-4f0e-a598-5e5cd855b294-workload-socket\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" Oct 30 13:26:50.923948 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:50.923925 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec336656-bd10-4f0e-a598-5e5cd855b294-istio-podinfo\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" Oct 30 13:26:51.031293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:51.025081 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec336656-bd10-4f0e-a598-5e5cd855b294-workload-certs\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" Oct 30 13:26:51.031293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:51.025160 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec336656-bd10-4f0e-a598-5e5cd855b294-istio-envoy\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" Oct 30 13:26:51.031293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:51.025190 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec336656-bd10-4f0e-a598-5e5cd855b294-istio-data\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" Oct 30 13:26:51.031293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:51.025218 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec336656-bd10-4f0e-a598-5e5cd855b294-istiod-ca-cert\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" Oct 30 13:26:51.031293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:51.025250 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec336656-bd10-4f0e-a598-5e5cd855b294-credential-socket\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" Oct 30 13:26:51.031293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:51.027674 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec336656-bd10-4f0e-a598-5e5cd855b294-istio-data\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" Oct 30 13:26:51.046907 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:51.034724 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec336656-bd10-4f0e-a598-5e5cd855b294-istio-token\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" Oct 30 13:26:51.046907 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:51.034829 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec336656-bd10-4f0e-a598-5e5cd855b294-workload-socket\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" Oct 30 13:26:51.046907 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:51.034874 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec336656-bd10-4f0e-a598-5e5cd855b294-istio-podinfo\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" Oct 30 13:26:51.046907 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:51.034948 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qss9\" (UniqueName: \"kubernetes.io/projected/ec336656-bd10-4f0e-a598-5e5cd855b294-kube-api-access-7qss9\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" Oct 30 13:26:51.046907 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:51.036242 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec336656-bd10-4f0e-a598-5e5cd855b294-credential-socket\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" Oct 30 13:26:51.046907 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:51.037846 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec336656-bd10-4f0e-a598-5e5cd855b294-istiod-ca-cert\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" Oct 30 13:26:51.046907 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:51.038093 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec336656-bd10-4f0e-a598-5e5cd855b294-workload-socket\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" Oct 30 13:26:51.046907 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:51.038450 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec336656-bd10-4f0e-a598-5e5cd855b294-workload-certs\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" Oct 30 13:26:51.046907 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:51.041356 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec336656-bd10-4f0e-a598-5e5cd855b294-istio-envoy\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" Oct 30 13:26:51.046907 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:51.041518 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec336656-bd10-4f0e-a598-5e5cd855b294-istio-podinfo\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" Oct 30 13:26:51.063345 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:51.063312 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec336656-bd10-4f0e-a598-5e5cd855b294-istio-token\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" Oct 30 13:26:51.063481 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:51.063443 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qss9\" (UniqueName: \"kubernetes.io/projected/ec336656-bd10-4f0e-a598-5e5cd855b294-kube-api-access-7qss9\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" Oct 30 13:26:51.103543 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:51.103518 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" Oct 30 13:26:51.438170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:51.438093 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn"] Oct 30 13:26:51.442698 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:26:51.442668 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec336656_bd10_4f0e_a598_5e5cd855b294.slice/crio-11735ca017a4667443b33217353abaf53db26c95bf8d8b941f3e6b00b767b892 WatchSource:0}: Error finding container 11735ca017a4667443b33217353abaf53db26c95bf8d8b941f3e6b00b767b892: Status 404 returned error can't find the container with id 11735ca017a4667443b33217353abaf53db26c95bf8d8b941f3e6b00b767b892 Oct 30 13:26:51.445890 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:51.445845 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:26:51.445982 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:51.445924 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:26:51.445982 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:51.445958 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:26:51.519639 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:51.519602 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" event={"ID":"ec336656-bd10-4f0e-a598-5e5cd855b294","Type":"ContainerStarted","Data":"11735ca017a4667443b33217353abaf53db26c95bf8d8b941f3e6b00b767b892"} Oct 30 13:26:51.656621 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:51.656591 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fc974f-9fcx4" Oct 30 13:26:51.843751 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:51.843716 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cbbc99e-cb4e-4388-a71f-fa4e87d6555e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cbbc99e-cb4e-4388-a71f-fa4e87d6555e-maas-api-xks-ca\") pod \"5cbbc99e-cb4e-4388-a71f-fa4e87d6555e\" (UID: \"5cbbc99e-cb4e-4388-a71f-fa4e87d6555e\") " Oct 30 13:26:51.843961 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:51.843826 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cbbc99e-cb4e-4388-a71f-fa4e87d6555e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cbbc99e-cb4e-4388-a71f-fa4e87d6555e-maas-api-openshift-ca\") pod \"5cbbc99e-cb4e-4388-a71f-fa4e87d6555e\" (UID: \"5cbbc99e-cb4e-4388-a71f-fa4e87d6555e\") " Oct 30 13:26:51.843961 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:51.843922 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cbbc99e-cb4e-4388-a71f-fa4e87d6555e-kube-api-access-762fj\" (UniqueName: \"kubernetes.io/projected/5cbbc99e-cb4e-4388-a71f-fa4e87d6555e-kube-api-access-762fj\") pod \"5cbbc99e-cb4e-4388-a71f-fa4e87d6555e\" (UID: \"5cbbc99e-cb4e-4388-a71f-fa4e87d6555e\") " Oct 30 13:26:51.844203 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:51.844174 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cbbc99e-cb4e-4388-a71f-fa4e87d6555e-maas-api-openshift-ca" pod "5cbbc99e-cb4e-4388-a71f-fa4e87d6555e" (UID: "5cbbc99e-cb4e-4388-a71f-fa4e87d6555e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:26:51.844523 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:51.844501 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cbbc99e-cb4e-4388-a71f-fa4e87d6555e-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:51.846546 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:51.846521 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cbbc99e-cb4e-4388-a71f-fa4e87d6555e-maas-api-xks-ca" pod "5cbbc99e-cb4e-4388-a71f-fa4e87d6555e" (UID: "5cbbc99e-cb4e-4388-a71f-fa4e87d6555e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:26:51.846546 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:51.846531 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cbbc99e-cb4e-4388-a71f-fa4e87d6555e-kube-api-access-762fj" pod "5cbbc99e-cb4e-4388-a71f-fa4e87d6555e" (UID: "5cbbc99e-cb4e-4388-a71f-fa4e87d6555e"). InnerVolumeSpecName "kube-api-access-762fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:51.945942 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:51.945911 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cbbc99e-cb4e-4388-a71f-fa4e87d6555e-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:51.945942 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:51.945939 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-762fj\" (UniqueName: \"kubernetes.io/projected/5cbbc99e-cb4e-4388-a71f-fa4e87d6555e-kube-api-access-762fj\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:52.526944 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:52.526895 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" event={"ID":"ec336656-bd10-4f0e-a598-5e5cd855b294","Type":"ContainerStarted","Data":"e374e69174c72d98eff478dfe35813c69cc3cd47cd59006020d4016126333ee3"} Oct 30 13:26:52.528355 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:52.528334 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fc974f-9fcx4" Oct 30 13:26:52.528465 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:52.528380 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fc974f-9fcx4" event={"ID":"5cbbc99e-cb4e-4388-a71f-fa4e87d6555e","Type":"ContainerDied","Data":"26071c6a9de4a15d2a7276bad1cf8c8d503cc9388ce7babfd3f2888f7c2311f5"} Oct 30 13:26:52.528465 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:52.528431 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26071c6a9de4a15d2a7276bad1cf8c8d503cc9388ce7babfd3f2888f7c2311f5" Oct 30 13:26:52.591843 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:52.591784 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" podStartSLOduration=2.591769931 podStartE2EDuration="2.591769931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:52.59123499 +0000 UTC m=+2511.194733435" watchObservedRunningTime="2026-10-30 13:26:52.591769931 +0000 UTC m=+2511.195268376" Oct 30 13:26:53.104423 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:53.104389 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" Oct 30 13:26:53.106409 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:53.106378 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 13:26:53.106569 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:53.106439 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" podUID="ec336656-bd10-4f0e-a598-5e5cd855b294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 13:26:54.104688 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:54.104562 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 13:26:54.104688 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:54.104615 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" podUID="ec336656-bd10-4f0e-a598-5e5cd855b294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 13:26:55.104325 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:55.104281 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 13:26:55.104561 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:55.104340 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" podUID="ec336656-bd10-4f0e-a598-5e5cd855b294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 13:26:55.277215 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:55.277172 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f3338e-krgmt"] Oct 30 13:26:55.277771 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:55.277756 4931 cpu_manager.go:414] "containerMap: removing container" podUID="5cbbc99e-cb4e-4388-a71f-fa4e87d6555e" containerName="revoke-keys" Oct 30 13:26:55.277771 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:55.277771 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cbbc99e-cb4e-4388-a71f-fa4e87d6555e" containerName="revoke-keys" Oct 30 13:26:55.277921 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:55.277910 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cbbc99e-cb4e-4388-a71f-fa4e87d6555e" containerName="revoke-keys" Oct 30 13:26:55.281743 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:55.281719 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f3338e-krgmt" Oct 30 13:26:55.305383 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:55.305347 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f3338e-krgmt"] Oct 30 13:26:55.314803 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:55.314766 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc1c51fe-68bf-4dbe-baee-2ab46607660a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f3338e-krgmt\" (UID: \"dc1c51fe-68bf-4dbe-baee-2ab46607660a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f3338e-krgmt" Oct 30 13:26:55.315038 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:55.315020 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc1c51fe-68bf-4dbe-baee-2ab46607660a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f3338e-krgmt\" (UID: \"dc1c51fe-68bf-4dbe-baee-2ab46607660a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f3338e-krgmt" Oct 30 13:26:55.315230 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:55.315217 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6srw\" (UniqueName: \"kubernetes.io/projected/dc1c51fe-68bf-4dbe-baee-2ab46607660a-kube-api-access-x6srw\") pod \"maas-api-revoke-keys-e2e-inf-b-f3338e-krgmt\" (UID: \"dc1c51fe-68bf-4dbe-baee-2ab46607660a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f3338e-krgmt" Oct 30 13:26:55.398136 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:55.398037 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:26:55.398136 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:55.398046 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:26:55.416964 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:55.416923 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc1c51fe-68bf-4dbe-baee-2ab46607660a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f3338e-krgmt\" (UID: \"dc1c51fe-68bf-4dbe-baee-2ab46607660a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f3338e-krgmt" Oct 30 13:26:55.417161 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:55.416985 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc1c51fe-68bf-4dbe-baee-2ab46607660a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f3338e-krgmt\" (UID: \"dc1c51fe-68bf-4dbe-baee-2ab46607660a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f3338e-krgmt" Oct 30 13:26:55.417161 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:55.417090 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6srw\" (UniqueName: \"kubernetes.io/projected/dc1c51fe-68bf-4dbe-baee-2ab46607660a-kube-api-access-x6srw\") pod \"maas-api-revoke-keys-e2e-inf-b-f3338e-krgmt\" (UID: \"dc1c51fe-68bf-4dbe-baee-2ab46607660a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f3338e-krgmt" Oct 30 13:26:55.417955 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:55.417926 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc1c51fe-68bf-4dbe-baee-2ab46607660a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f3338e-krgmt\" (UID: \"dc1c51fe-68bf-4dbe-baee-2ab46607660a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f3338e-krgmt" Oct 30 13:26:55.419774 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:55.419753 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc1c51fe-68bf-4dbe-baee-2ab46607660a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f3338e-krgmt\" (UID: \"dc1c51fe-68bf-4dbe-baee-2ab46607660a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f3338e-krgmt" Oct 30 13:26:55.462063 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:55.462030 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6srw\" (UniqueName: \"kubernetes.io/projected/dc1c51fe-68bf-4dbe-baee-2ab46607660a-kube-api-access-x6srw\") pod \"maas-api-revoke-keys-e2e-inf-b-f3338e-krgmt\" (UID: \"dc1c51fe-68bf-4dbe-baee-2ab46607660a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f3338e-krgmt" Oct 30 13:26:55.595965 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:55.595927 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f3338e-krgmt" Oct 30 13:26:56.105034 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:56.105001 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 13:26:56.105213 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:56.105052 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" podUID="ec336656-bd10-4f0e-a598-5e5cd855b294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 13:26:56.158296 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:56.149514 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f3338e-krgmt"] Oct 30 13:26:56.162290 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:26:56.161797 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc1c51fe_68bf_4dbe_baee_2ab46607660a.slice/crio-68f71cc0ac44b76bb655d5f81fcce8c466924cfcd19352a7fc0ff72f40438d00 WatchSource:0}: Error finding container 68f71cc0ac44b76bb655d5f81fcce8c466924cfcd19352a7fc0ff72f40438d00: Status 404 returned error can't find the container with id 68f71cc0ac44b76bb655d5f81fcce8c466924cfcd19352a7fc0ff72f40438d00 Oct 30 13:26:56.583239 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:56.583196 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f3338e-krgmt" event={"ID":"dc1c51fe-68bf-4dbe-baee-2ab46607660a","Type":"ContainerStarted","Data":"68f71cc0ac44b76bb655d5f81fcce8c466924cfcd19352a7fc0ff72f40438d00"} Oct 30 13:26:56.842849 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:56.842742 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:26:57.104209 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:57.104127 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 13:26:57.104209 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:57.104182 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" podUID="ec336656-bd10-4f0e-a598-5e5cd855b294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 13:26:57.595071 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:57.595032 4931 generic.go:356] "Generic (PLEG): container finished" podID="dc1c51fe-68bf-4dbe-baee-2ab46607660a" containerID="c135dd7d09767db66e07ab2581bb159c0905d432a0a7f81f4a3b6c3a40d3107d" exitCode=0 Oct 30 13:26:57.595697 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:57.595466 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f3338e-krgmt" event={"ID":"dc1c51fe-68bf-4dbe-baee-2ab46607660a","Type":"ContainerDied","Data":"c135dd7d09767db66e07ab2581bb159c0905d432a0a7f81f4a3b6c3a40d3107d"} Oct 30 13:26:58.106319 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:58.104764 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 13:26:58.106319 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:58.104866 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" podUID="ec336656-bd10-4f0e-a598-5e5cd855b294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 13:26:58.796046 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:58.796018 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f3338e-krgmt" Oct 30 13:26:58.928203 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:58.928114 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc1c51fe-68bf-4dbe-baee-2ab46607660a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc1c51fe-68bf-4dbe-baee-2ab46607660a-maas-api-openshift-ca\") pod \"dc1c51fe-68bf-4dbe-baee-2ab46607660a\" (UID: \"dc1c51fe-68bf-4dbe-baee-2ab46607660a\") " Oct 30 13:26:58.928203 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:58.928188 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc1c51fe-68bf-4dbe-baee-2ab46607660a-kube-api-access-x6srw\" (UniqueName: \"kubernetes.io/projected/dc1c51fe-68bf-4dbe-baee-2ab46607660a-kube-api-access-x6srw\") pod \"dc1c51fe-68bf-4dbe-baee-2ab46607660a\" (UID: \"dc1c51fe-68bf-4dbe-baee-2ab46607660a\") " Oct 30 13:26:58.928559 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:58.928241 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc1c51fe-68bf-4dbe-baee-2ab46607660a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc1c51fe-68bf-4dbe-baee-2ab46607660a-maas-api-xks-ca\") pod \"dc1c51fe-68bf-4dbe-baee-2ab46607660a\" (UID: \"dc1c51fe-68bf-4dbe-baee-2ab46607660a\") " Oct 30 13:26:58.928626 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:58.928590 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc1c51fe-68bf-4dbe-baee-2ab46607660a-maas-api-openshift-ca" pod "dc1c51fe-68bf-4dbe-baee-2ab46607660a" (UID: "dc1c51fe-68bf-4dbe-baee-2ab46607660a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:26:58.928806 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:58.928788 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc1c51fe-68bf-4dbe-baee-2ab46607660a-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:58.930537 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:58.930498 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc1c51fe-68bf-4dbe-baee-2ab46607660a-maas-api-xks-ca" pod "dc1c51fe-68bf-4dbe-baee-2ab46607660a" (UID: "dc1c51fe-68bf-4dbe-baee-2ab46607660a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:26:58.930663 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:58.930639 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc1c51fe-68bf-4dbe-baee-2ab46607660a-kube-api-access-x6srw" pod "dc1c51fe-68bf-4dbe-baee-2ab46607660a" (UID: "dc1c51fe-68bf-4dbe-baee-2ab46607660a"). InnerVolumeSpecName "kube-api-access-x6srw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:59.030077 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:59.030041 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6srw\" (UniqueName: \"kubernetes.io/projected/dc1c51fe-68bf-4dbe-baee-2ab46607660a-kube-api-access-x6srw\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:59.030077 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:59.030073 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc1c51fe-68bf-4dbe-baee-2ab46607660a-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:26:59.108917 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:59.108885 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" Oct 30 13:26:59.109383 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:59.109363 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" Oct 30 13:26:59.110232 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:59.110212 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" Oct 30 13:26:59.623668 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:59.623629 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f3338e-krgmt" event={"ID":"dc1c51fe-68bf-4dbe-baee-2ab46607660a","Type":"ContainerDied","Data":"68f71cc0ac44b76bb655d5f81fcce8c466924cfcd19352a7fc0ff72f40438d00"} Oct 30 13:26:59.623869 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:59.623689 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68f71cc0ac44b76bb655d5f81fcce8c466924cfcd19352a7fc0ff72f40438d00" Oct 30 13:26:59.623869 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:26:59.623648 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f3338e-krgmt" Oct 30 13:27:03.583970 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:03.583890 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a71c333e-9bxjf"] Oct 30 13:27:03.584569 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:03.584551 4931 cpu_manager.go:414] "containerMap: removing container" podUID="dc1c51fe-68bf-4dbe-baee-2ab46607660a" containerName="revoke-keys" Oct 30 13:27:03.584619 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:03.584571 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc1c51fe-68bf-4dbe-baee-2ab46607660a" containerName="revoke-keys" Oct 30 13:27:03.584679 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:03.584669 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc1c51fe-68bf-4dbe-baee-2ab46607660a" containerName="revoke-keys" Oct 30 13:27:03.589241 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:03.589223 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71c333e-9bxjf" Oct 30 13:27:03.612039 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:03.612008 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:27:03.620277 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:03.620237 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:27:03.624647 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:03.624621 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a71c333e-9bxjf"] Oct 30 13:27:03.697367 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:03.697329 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ef3c3b3-766a-4eda-9734-819f35478218-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a71c333e-9bxjf\" (UID: \"4ef3c3b3-766a-4eda-9734-819f35478218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71c333e-9bxjf" Oct 30 13:27:03.697367 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:03.697370 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs7s9\" (UniqueName: \"kubernetes.io/projected/4ef3c3b3-766a-4eda-9734-819f35478218-kube-api-access-qs7s9\") pod \"maas-api-revoke-sub-a71c333e-9bxjf\" (UID: \"4ef3c3b3-766a-4eda-9734-819f35478218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71c333e-9bxjf" Oct 30 13:27:03.697603 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:03.697582 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ef3c3b3-766a-4eda-9734-819f35478218-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a71c333e-9bxjf\" (UID: \"4ef3c3b3-766a-4eda-9734-819f35478218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71c333e-9bxjf" Oct 30 13:27:03.798425 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:03.798387 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ef3c3b3-766a-4eda-9734-819f35478218-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a71c333e-9bxjf\" (UID: \"4ef3c3b3-766a-4eda-9734-819f35478218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71c333e-9bxjf" Oct 30 13:27:03.798635 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:03.798544 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ef3c3b3-766a-4eda-9734-819f35478218-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a71c333e-9bxjf\" (UID: \"4ef3c3b3-766a-4eda-9734-819f35478218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71c333e-9bxjf" Oct 30 13:27:03.798635 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:03.798573 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs7s9\" (UniqueName: \"kubernetes.io/projected/4ef3c3b3-766a-4eda-9734-819f35478218-kube-api-access-qs7s9\") pod \"maas-api-revoke-sub-a71c333e-9bxjf\" (UID: \"4ef3c3b3-766a-4eda-9734-819f35478218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71c333e-9bxjf" Oct 30 13:27:03.799218 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:03.799194 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ef3c3b3-766a-4eda-9734-819f35478218-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a71c333e-9bxjf\" (UID: \"4ef3c3b3-766a-4eda-9734-819f35478218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71c333e-9bxjf" Oct 30 13:27:03.800956 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:03.800933 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ef3c3b3-766a-4eda-9734-819f35478218-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a71c333e-9bxjf\" (UID: \"4ef3c3b3-766a-4eda-9734-819f35478218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71c333e-9bxjf" Oct 30 13:27:03.825025 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:03.824999 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs7s9\" (UniqueName: \"kubernetes.io/projected/4ef3c3b3-766a-4eda-9734-819f35478218-kube-api-access-qs7s9\") pod \"maas-api-revoke-sub-a71c333e-9bxjf\" (UID: \"4ef3c3b3-766a-4eda-9734-819f35478218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71c333e-9bxjf" Oct 30 13:27:03.903304 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:03.903194 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71c333e-9bxjf" Oct 30 13:27:04.222343 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:04.222310 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a71c333e-9bxjf"] Oct 30 13:27:04.222767 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:27:04.222736 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ef3c3b3_766a_4eda_9734_819f35478218.slice/crio-32eaedf0c0bf9e58d576bf6d44817db69d819ccce0dfd12229d0374bde9280dd WatchSource:0}: Error finding container 32eaedf0c0bf9e58d576bf6d44817db69d819ccce0dfd12229d0374bde9280dd: Status 404 returned error can't find the container with id 32eaedf0c0bf9e58d576bf6d44817db69d819ccce0dfd12229d0374bde9280dd Oct 30 13:27:04.663063 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:04.663016 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71c333e-9bxjf" event={"ID":"4ef3c3b3-766a-4eda-9734-819f35478218","Type":"ContainerStarted","Data":"23b0cbcb6bd270adcf5161327e93d8d1cca3546f56b0752356f65e228e3128e9"} Oct 30 13:27:04.663556 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:04.663083 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71c333e-9bxjf" event={"ID":"4ef3c3b3-766a-4eda-9734-819f35478218","Type":"ContainerStarted","Data":"32eaedf0c0bf9e58d576bf6d44817db69d819ccce0dfd12229d0374bde9280dd"} Oct 30 13:27:04.713915 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:04.713853 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71c333e-9bxjf" podStartSLOduration=1.713838783 podStartE2EDuration="1.713838783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:04.710340839 +0000 UTC m=+2523.313839284" watchObservedRunningTime="2026-10-30 13:27:04.713838783 +0000 UTC m=+2523.317337228" Oct 30 13:27:05.670968 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:05.670934 4931 generic.go:356] "Generic (PLEG): container finished" podID="4ef3c3b3-766a-4eda-9734-819f35478218" containerID="23b0cbcb6bd270adcf5161327e93d8d1cca3546f56b0752356f65e228e3128e9" exitCode=0 Oct 30 13:27:05.671501 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:05.671020 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71c333e-9bxjf" event={"ID":"4ef3c3b3-766a-4eda-9734-819f35478218","Type":"ContainerDied","Data":"23b0cbcb6bd270adcf5161327e93d8d1cca3546f56b0752356f65e228e3128e9"} Oct 30 13:27:06.850089 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:06.850060 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71c333e-9bxjf" Oct 30 13:27:06.866204 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:06.866168 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ef3c3b3-766a-4eda-9734-819f35478218-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ef3c3b3-766a-4eda-9734-819f35478218-maas-api-xks-ca\") pod \"4ef3c3b3-766a-4eda-9734-819f35478218\" (UID: \"4ef3c3b3-766a-4eda-9734-819f35478218\") " Oct 30 13:27:06.866395 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:06.866253 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ef3c3b3-766a-4eda-9734-819f35478218-kube-api-access-qs7s9\" (UniqueName: \"kubernetes.io/projected/4ef3c3b3-766a-4eda-9734-819f35478218-kube-api-access-qs7s9\") pod \"4ef3c3b3-766a-4eda-9734-819f35478218\" (UID: \"4ef3c3b3-766a-4eda-9734-819f35478218\") " Oct 30 13:27:06.866395 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:06.866368 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ef3c3b3-766a-4eda-9734-819f35478218-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ef3c3b3-766a-4eda-9734-819f35478218-maas-api-openshift-ca\") pod \"4ef3c3b3-766a-4eda-9734-819f35478218\" (UID: \"4ef3c3b3-766a-4eda-9734-819f35478218\") " Oct 30 13:27:06.866755 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:06.866723 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ef3c3b3-766a-4eda-9734-819f35478218-maas-api-openshift-ca" pod "4ef3c3b3-766a-4eda-9734-819f35478218" (UID: "4ef3c3b3-766a-4eda-9734-819f35478218"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:27:06.867133 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:06.867107 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ef3c3b3-766a-4eda-9734-819f35478218-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:27:06.868631 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:06.868601 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ef3c3b3-766a-4eda-9734-819f35478218-maas-api-xks-ca" pod "4ef3c3b3-766a-4eda-9734-819f35478218" (UID: "4ef3c3b3-766a-4eda-9734-819f35478218"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:27:06.868729 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:06.868639 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ef3c3b3-766a-4eda-9734-819f35478218-kube-api-access-qs7s9" pod "4ef3c3b3-766a-4eda-9734-819f35478218" (UID: "4ef3c3b3-766a-4eda-9734-819f35478218"). InnerVolumeSpecName "kube-api-access-qs7s9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:06.968461 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:06.968419 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ef3c3b3-766a-4eda-9734-819f35478218-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:27:06.968461 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:06.968455 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs7s9\" (UniqueName: \"kubernetes.io/projected/4ef3c3b3-766a-4eda-9734-819f35478218-kube-api-access-qs7s9\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:27:07.352992 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:27:07.352904 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:27:07.688655 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:07.688587 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71c333e-9bxjf" Oct 30 13:27:07.688655 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:07.688626 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71c333e-9bxjf" event={"ID":"4ef3c3b3-766a-4eda-9734-819f35478218","Type":"ContainerDied","Data":"32eaedf0c0bf9e58d576bf6d44817db69d819ccce0dfd12229d0374bde9280dd"} Oct 30 13:27:07.688844 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:07.688673 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32eaedf0c0bf9e58d576bf6d44817db69d819ccce0dfd12229d0374bde9280dd" Oct 30 13:27:17.519706 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:17.519669 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b8461-5jqqv"] Oct 30 13:27:17.520201 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:17.520187 4931 cpu_manager.go:414] "containerMap: removing container" podUID="4ef3c3b3-766a-4eda-9734-819f35478218" containerName="revoke-keys" Oct 30 13:27:17.520245 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:17.520202 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ef3c3b3-766a-4eda-9734-819f35478218" containerName="revoke-keys" Oct 30 13:27:17.520344 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:17.520333 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ef3c3b3-766a-4eda-9734-819f35478218" containerName="revoke-keys" Oct 30 13:27:17.523817 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:17.523798 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b8461-5jqqv" Oct 30 13:27:17.551485 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:17.551451 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b8461-5jqqv"] Oct 30 13:27:17.557363 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:17.557334 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:27:17.557501 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:17.557404 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:27:17.619262 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:17.619219 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/914e1460-a166-4f84-8f6a-f84589bda525-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2b8461-5jqqv\" (UID: \"914e1460-a166-4f84-8f6a-f84589bda525\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b8461-5jqqv" Oct 30 13:27:17.619471 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:17.619373 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/914e1460-a166-4f84-8f6a-f84589bda525-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2b8461-5jqqv\" (UID: \"914e1460-a166-4f84-8f6a-f84589bda525\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b8461-5jqqv" Oct 30 13:27:17.619570 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:17.619538 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvlmc\" (UniqueName: \"kubernetes.io/projected/914e1460-a166-4f84-8f6a-f84589bda525-kube-api-access-qvlmc\") pod \"maas-api-revoke-keys-e2e-inf-a-2b8461-5jqqv\" (UID: \"914e1460-a166-4f84-8f6a-f84589bda525\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b8461-5jqqv" Oct 30 13:27:17.720437 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:17.720403 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvlmc\" (UniqueName: \"kubernetes.io/projected/914e1460-a166-4f84-8f6a-f84589bda525-kube-api-access-qvlmc\") pod \"maas-api-revoke-keys-e2e-inf-a-2b8461-5jqqv\" (UID: \"914e1460-a166-4f84-8f6a-f84589bda525\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b8461-5jqqv" Oct 30 13:27:17.720643 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:17.720612 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/914e1460-a166-4f84-8f6a-f84589bda525-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2b8461-5jqqv\" (UID: \"914e1460-a166-4f84-8f6a-f84589bda525\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b8461-5jqqv" Oct 30 13:27:17.720716 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:17.720664 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/914e1460-a166-4f84-8f6a-f84589bda525-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2b8461-5jqqv\" (UID: \"914e1460-a166-4f84-8f6a-f84589bda525\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b8461-5jqqv" Oct 30 13:27:17.721434 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:17.721409 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/914e1460-a166-4f84-8f6a-f84589bda525-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2b8461-5jqqv\" (UID: \"914e1460-a166-4f84-8f6a-f84589bda525\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b8461-5jqqv" Oct 30 13:27:17.723066 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:17.723039 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/914e1460-a166-4f84-8f6a-f84589bda525-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2b8461-5jqqv\" (UID: \"914e1460-a166-4f84-8f6a-f84589bda525\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b8461-5jqqv" Oct 30 13:27:17.744744 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:17.744715 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvlmc\" (UniqueName: \"kubernetes.io/projected/914e1460-a166-4f84-8f6a-f84589bda525-kube-api-access-qvlmc\") pod \"maas-api-revoke-keys-e2e-inf-a-2b8461-5jqqv\" (UID: \"914e1460-a166-4f84-8f6a-f84589bda525\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b8461-5jqqv" Oct 30 13:27:17.839051 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:17.838972 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b8461-5jqqv" Oct 30 13:27:18.145077 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:18.145052 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b8461-5jqqv"] Oct 30 13:27:18.145938 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:27:18.145904 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod914e1460_a166_4f84_8f6a_f84589bda525.slice/crio-278cfa2dec1b4aa3f833009dbbc3a89e8b1fed7d1a7d8db1aa4039f636081d6d WatchSource:0}: Error finding container 278cfa2dec1b4aa3f833009dbbc3a89e8b1fed7d1a7d8db1aa4039f636081d6d: Status 404 returned error can't find the container with id 278cfa2dec1b4aa3f833009dbbc3a89e8b1fed7d1a7d8db1aa4039f636081d6d Oct 30 13:27:18.771027 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:18.770937 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b8461-5jqqv" event={"ID":"914e1460-a166-4f84-8f6a-f84589bda525","Type":"ContainerStarted","Data":"11ca4a3af61346d91d71de9c629fcb0ca70c8fa8b50bff3898a1d1f842b2b977"} Oct 30 13:27:18.771027 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:18.770997 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b8461-5jqqv" event={"ID":"914e1460-a166-4f84-8f6a-f84589bda525","Type":"ContainerStarted","Data":"278cfa2dec1b4aa3f833009dbbc3a89e8b1fed7d1a7d8db1aa4039f636081d6d"} Oct 30 13:27:18.817462 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:18.817410 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b8461-5jqqv" podStartSLOduration=1.81739742 podStartE2EDuration="1.81739742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:18.815994301 +0000 UTC m=+2537.419492748" watchObservedRunningTime="2026-10-30 13:27:18.81739742 +0000 UTC m=+2537.420895866" Oct 30 13:27:19.783369 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:19.783318 4931 generic.go:356] "Generic (PLEG): container finished" podID="914e1460-a166-4f84-8f6a-f84589bda525" containerID="11ca4a3af61346d91d71de9c629fcb0ca70c8fa8b50bff3898a1d1f842b2b977" exitCode=0 Oct 30 13:27:19.783894 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:19.783528 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b8461-5jqqv" event={"ID":"914e1460-a166-4f84-8f6a-f84589bda525","Type":"ContainerDied","Data":"11ca4a3af61346d91d71de9c629fcb0ca70c8fa8b50bff3898a1d1f842b2b977"} Oct 30 13:27:20.947728 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:20.947699 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b8461-5jqqv" Oct 30 13:27:21.067130 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:21.067048 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/914e1460-a166-4f84-8f6a-f84589bda525-kube-api-access-qvlmc\" (UniqueName: \"kubernetes.io/projected/914e1460-a166-4f84-8f6a-f84589bda525-kube-api-access-qvlmc\") pod \"914e1460-a166-4f84-8f6a-f84589bda525\" (UID: \"914e1460-a166-4f84-8f6a-f84589bda525\") " Oct 30 13:27:21.067130 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:21.067093 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/914e1460-a166-4f84-8f6a-f84589bda525-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/914e1460-a166-4f84-8f6a-f84589bda525-maas-api-xks-ca\") pod \"914e1460-a166-4f84-8f6a-f84589bda525\" (UID: \"914e1460-a166-4f84-8f6a-f84589bda525\") " Oct 30 13:27:21.067363 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:21.067206 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/914e1460-a166-4f84-8f6a-f84589bda525-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/914e1460-a166-4f84-8f6a-f84589bda525-maas-api-openshift-ca\") pod \"914e1460-a166-4f84-8f6a-f84589bda525\" (UID: \"914e1460-a166-4f84-8f6a-f84589bda525\") " Oct 30 13:27:21.067698 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:21.067667 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/914e1460-a166-4f84-8f6a-f84589bda525-maas-api-openshift-ca" pod "914e1460-a166-4f84-8f6a-f84589bda525" (UID: "914e1460-a166-4f84-8f6a-f84589bda525"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:27:21.067938 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:21.067905 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/914e1460-a166-4f84-8f6a-f84589bda525-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:27:21.069556 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:21.069527 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/914e1460-a166-4f84-8f6a-f84589bda525-maas-api-xks-ca" pod "914e1460-a166-4f84-8f6a-f84589bda525" (UID: "914e1460-a166-4f84-8f6a-f84589bda525"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:27:21.069639 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:21.069618 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/914e1460-a166-4f84-8f6a-f84589bda525-kube-api-access-qvlmc" pod "914e1460-a166-4f84-8f6a-f84589bda525" (UID: "914e1460-a166-4f84-8f6a-f84589bda525"). InnerVolumeSpecName "kube-api-access-qvlmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:21.169030 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:21.168998 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvlmc\" (UniqueName: \"kubernetes.io/projected/914e1460-a166-4f84-8f6a-f84589bda525-kube-api-access-qvlmc\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:27:21.169030 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:21.169029 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/914e1460-a166-4f84-8f6a-f84589bda525-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:27:21.800373 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:21.800299 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b8461-5jqqv" Oct 30 13:27:21.800523 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:21.800299 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b8461-5jqqv" event={"ID":"914e1460-a166-4f84-8f6a-f84589bda525","Type":"ContainerDied","Data":"278cfa2dec1b4aa3f833009dbbc3a89e8b1fed7d1a7d8db1aa4039f636081d6d"} Oct 30 13:27:21.800523 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:21.800421 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="278cfa2dec1b4aa3f833009dbbc3a89e8b1fed7d1a7d8db1aa4039f636081d6d" Oct 30 13:27:37.353885 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:27:37.353837 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:27:39.363600 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:39.363560 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f3338e-krgmt"] Oct 30 13:27:39.376913 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:39.376872 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f3338e-krgmt"] Oct 30 13:27:39.531596 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:39.531541 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc1c51fe-68bf-4dbe-baee-2ab46607660a" path="/var/lib/kubelet/pods/dc1c51fe-68bf-4dbe-baee-2ab46607660a/volumes" Oct 30 13:27:43.592912 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:43.592820 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm"] Oct 30 13:27:43.593647 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:43.593609 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm" podUID="7a109d1f-9e17-4638-a451-54d814f34fbd" containerName="maas-api" containerID="cri-o://9b0ed253e5e5eda6f0f0db8bf81d4f8e2467e85f8e5a4732e10abc3f19d40320" gracePeriod=30 Oct 30 13:27:43.979163 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:43.979128 4931 generic.go:356] "Generic (PLEG): container finished" podID="7a109d1f-9e17-4638-a451-54d814f34fbd" containerID="9b0ed253e5e5eda6f0f0db8bf81d4f8e2467e85f8e5a4732e10abc3f19d40320" exitCode=0 Oct 30 13:27:43.979397 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:43.979199 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm" event={"ID":"7a109d1f-9e17-4638-a451-54d814f34fbd","Type":"ContainerDied","Data":"9b0ed253e5e5eda6f0f0db8bf81d4f8e2467e85f8e5a4732e10abc3f19d40320"} Oct 30 13:27:44.256204 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:44.256175 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm" Oct 30 13:27:44.313066 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:44.313035 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a109d1f-9e17-4638-a451-54d814f34fbd-kube-api-access-zjzw8\" (UniqueName: \"kubernetes.io/projected/7a109d1f-9e17-4638-a451-54d814f34fbd-kube-api-access-zjzw8\") pod \"7a109d1f-9e17-4638-a451-54d814f34fbd\" (UID: \"7a109d1f-9e17-4638-a451-54d814f34fbd\") " Oct 30 13:27:44.313228 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:44.313185 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a109d1f-9e17-4638-a451-54d814f34fbd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a109d1f-9e17-4638-a451-54d814f34fbd-metrics-certs\") pod \"7a109d1f-9e17-4638-a451-54d814f34fbd\" (UID: \"7a109d1f-9e17-4638-a451-54d814f34fbd\") " Oct 30 13:27:44.313331 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:44.313317 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a109d1f-9e17-4638-a451-54d814f34fbd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a109d1f-9e17-4638-a451-54d814f34fbd-openshift-service-ca\") pod \"7a109d1f-9e17-4638-a451-54d814f34fbd\" (UID: \"7a109d1f-9e17-4638-a451-54d814f34fbd\") " Oct 30 13:27:44.313408 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:44.313339 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a109d1f-9e17-4638-a451-54d814f34fbd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a109d1f-9e17-4638-a451-54d814f34fbd-maas-api-tls\") pod \"7a109d1f-9e17-4638-a451-54d814f34fbd\" (UID: \"7a109d1f-9e17-4638-a451-54d814f34fbd\") " Oct 30 13:27:44.313746 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:44.313698 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a109d1f-9e17-4638-a451-54d814f34fbd-openshift-service-ca" pod "7a109d1f-9e17-4638-a451-54d814f34fbd" (UID: "7a109d1f-9e17-4638-a451-54d814f34fbd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:27:44.313900 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:44.313872 4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a109d1f-9e17-4638-a451-54d814f34fbd-openshift-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:27:44.315590 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:44.315560 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a109d1f-9e17-4638-a451-54d814f34fbd-kube-api-access-zjzw8" pod "7a109d1f-9e17-4638-a451-54d814f34fbd" (UID: "7a109d1f-9e17-4638-a451-54d814f34fbd"). InnerVolumeSpecName "kube-api-access-zjzw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:44.317172 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:44.317138 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a109d1f-9e17-4638-a451-54d814f34fbd-maas-api-tls" pod "7a109d1f-9e17-4638-a451-54d814f34fbd" (UID: "7a109d1f-9e17-4638-a451-54d814f34fbd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:27:44.317423 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:44.317396 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a109d1f-9e17-4638-a451-54d814f34fbd-metrics-certs" pod "7a109d1f-9e17-4638-a451-54d814f34fbd" (UID: "7a109d1f-9e17-4638-a451-54d814f34fbd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:27:44.415041 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:44.415008 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a109d1f-9e17-4638-a451-54d814f34fbd-maas-api-tls\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:27:44.415245 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:44.415054 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjzw8\" (UniqueName: \"kubernetes.io/projected/7a109d1f-9e17-4638-a451-54d814f34fbd-kube-api-access-zjzw8\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:27:44.415245 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:44.415065 4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a109d1f-9e17-4638-a451-54d814f34fbd-metrics-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:27:44.741162 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:44.741125 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b8461-5jqqv"] Oct 30 13:27:44.746947 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:44.746921 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b8461-5jqqv"] Oct 30 13:27:45.058766 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:45.058610 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm" event={"ID":"7a109d1f-9e17-4638-a451-54d814f34fbd","Type":"ContainerDied","Data":"8fcf892992db13d197166771faa810f2062cd94630633cea61f69cc1874e795e"} Oct 30 13:27:45.058766 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:45.058687 4931 scope.go:122] "RemoveContainer" containerID="9b0ed253e5e5eda6f0f0db8bf81d4f8e2467e85f8e5a4732e10abc3f19d40320" Oct 30 13:27:45.059004 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:45.058965 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm" Oct 30 13:27:45.137772 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:45.137736 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm"] Oct 30 13:27:45.141488 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:45.141466 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2b8461-75bbd4666d-hsvdm"] Oct 30 13:27:45.515060 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:45.515027 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a109d1f-9e17-4638-a451-54d814f34fbd" path="/var/lib/kubelet/pods/7a109d1f-9e17-4638-a451-54d814f34fbd/volumes" Oct 30 13:27:45.515565 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:45.515545 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="914e1460-a166-4f84-8f6a-f84589bda525" path="/var/lib/kubelet/pods/914e1460-a166-4f84-8f6a-f84589bda525/volumes" Oct 30 13:27:47.246743 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:47.246703 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k"] Oct 30 13:27:47.247183 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:47.246916 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" podUID="f314d798-3b21-4d5d-9f42-e1eee98c647c" containerName="istio-proxy" containerID="cri-o://c5ee9c9e0b15271c1ad7397bbd81d05bb3843220349bf8028af678ccd3cf16e1" gracePeriod=30 Oct 30 13:27:47.529494 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:47.529412 4931 patch_prober.go:29] interesting pod/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:27:47.529494 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:47.529478 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" podUID="f314d798-3b21-4d5d-9f42-e1eee98c647c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:27:52.533177 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.533118 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:27:52.627818 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.627788 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f314d798-3b21-4d5d-9f42-e1eee98c647c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f314d798-3b21-4d5d-9f42-e1eee98c647c-credential-socket\") pod \"f314d798-3b21-4d5d-9f42-e1eee98c647c\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " Oct 30 13:27:52.628003 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.627861 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f314d798-3b21-4d5d-9f42-e1eee98c647c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f314d798-3b21-4d5d-9f42-e1eee98c647c-istio-data\") pod \"f314d798-3b21-4d5d-9f42-e1eee98c647c\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " Oct 30 13:27:52.628003 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.627891 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f314d798-3b21-4d5d-9f42-e1eee98c647c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f314d798-3b21-4d5d-9f42-e1eee98c647c-istio-envoy\") pod \"f314d798-3b21-4d5d-9f42-e1eee98c647c\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " Oct 30 13:27:52.628003 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.627951 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f314d798-3b21-4d5d-9f42-e1eee98c647c-kube-api-access-rg7vd\" (UniqueName: \"kubernetes.io/projected/f314d798-3b21-4d5d-9f42-e1eee98c647c-kube-api-access-rg7vd\") pod \"f314d798-3b21-4d5d-9f42-e1eee98c647c\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " Oct 30 13:27:52.628003 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.627997 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f314d798-3b21-4d5d-9f42-e1eee98c647c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f314d798-3b21-4d5d-9f42-e1eee98c647c-workload-socket\") pod \"f314d798-3b21-4d5d-9f42-e1eee98c647c\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " Oct 30 13:27:52.628251 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.628050 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f314d798-3b21-4d5d-9f42-e1eee98c647c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f314d798-3b21-4d5d-9f42-e1eee98c647c-workload-certs\") pod \"f314d798-3b21-4d5d-9f42-e1eee98c647c\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " Oct 30 13:27:52.628251 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.628087 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f314d798-3b21-4d5d-9f42-e1eee98c647c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f314d798-3b21-4d5d-9f42-e1eee98c647c-istiod-ca-cert\") pod \"f314d798-3b21-4d5d-9f42-e1eee98c647c\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " Oct 30 13:27:52.628251 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.628089 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f314d798-3b21-4d5d-9f42-e1eee98c647c-credential-socket" pod "f314d798-3b21-4d5d-9f42-e1eee98c647c" (UID: "f314d798-3b21-4d5d-9f42-e1eee98c647c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:52.628251 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.628167 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f314d798-3b21-4d5d-9f42-e1eee98c647c-istio-data" pod "f314d798-3b21-4d5d-9f42-e1eee98c647c" (UID: "f314d798-3b21-4d5d-9f42-e1eee98c647c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:52.628251 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.628178 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f314d798-3b21-4d5d-9f42-e1eee98c647c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f314d798-3b21-4d5d-9f42-e1eee98c647c-istio-podinfo\") pod \"f314d798-3b21-4d5d-9f42-e1eee98c647c\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " Oct 30 13:27:52.628251 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.628218 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f314d798-3b21-4d5d-9f42-e1eee98c647c-istio-token\" (UniqueName: \"kubernetes.io/projected/f314d798-3b21-4d5d-9f42-e1eee98c647c-istio-token\") pod \"f314d798-3b21-4d5d-9f42-e1eee98c647c\" (UID: \"f314d798-3b21-4d5d-9f42-e1eee98c647c\") " Oct 30 13:27:52.628607 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.628345 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f314d798-3b21-4d5d-9f42-e1eee98c647c-workload-certs" pod "f314d798-3b21-4d5d-9f42-e1eee98c647c" (UID: "f314d798-3b21-4d5d-9f42-e1eee98c647c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:52.628607 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.628554 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f314d798-3b21-4d5d-9f42-e1eee98c647c-istiod-ca-cert" pod "f314d798-3b21-4d5d-9f42-e1eee98c647c" (UID: "f314d798-3b21-4d5d-9f42-e1eee98c647c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:27:52.628976 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.628925 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f314d798-3b21-4d5d-9f42-e1eee98c647c-workload-socket" pod "f314d798-3b21-4d5d-9f42-e1eee98c647c" (UID: "f314d798-3b21-4d5d-9f42-e1eee98c647c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:52.629083 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.629034 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f314d798-3b21-4d5d-9f42-e1eee98c647c-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:27:52.629083 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.629057 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f314d798-3b21-4d5d-9f42-e1eee98c647c-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:27:52.629083 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.629074 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f314d798-3b21-4d5d-9f42-e1eee98c647c-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:27:52.629257 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.629088 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f314d798-3b21-4d5d-9f42-e1eee98c647c-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:27:52.630457 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.630419 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f314d798-3b21-4d5d-9f42-e1eee98c647c-istio-envoy" pod "f314d798-3b21-4d5d-9f42-e1eee98c647c" (UID: "f314d798-3b21-4d5d-9f42-e1eee98c647c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:52.630726 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.630704 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f314d798-3b21-4d5d-9f42-e1eee98c647c-istio-podinfo" pod "f314d798-3b21-4d5d-9f42-e1eee98c647c" (UID: "f314d798-3b21-4d5d-9f42-e1eee98c647c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:27:52.630958 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.630938 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f314d798-3b21-4d5d-9f42-e1eee98c647c-kube-api-access-rg7vd" pod "f314d798-3b21-4d5d-9f42-e1eee98c647c" (UID: "f314d798-3b21-4d5d-9f42-e1eee98c647c"). InnerVolumeSpecName "kube-api-access-rg7vd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:52.631035 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.630972 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f314d798-3b21-4d5d-9f42-e1eee98c647c-istio-token" pod "f314d798-3b21-4d5d-9f42-e1eee98c647c" (UID: "f314d798-3b21-4d5d-9f42-e1eee98c647c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:52.730664 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.730632 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f314d798-3b21-4d5d-9f42-e1eee98c647c-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:27:52.730664 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.730660 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg7vd\" (UniqueName: \"kubernetes.io/projected/f314d798-3b21-4d5d-9f42-e1eee98c647c-kube-api-access-rg7vd\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:27:52.730664 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.730671 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f314d798-3b21-4d5d-9f42-e1eee98c647c-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:27:52.730911 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.730680 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f314d798-3b21-4d5d-9f42-e1eee98c647c-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:27:52.730911 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.730689 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f314d798-3b21-4d5d-9f42-e1eee98c647c-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:27:52.753971 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.753927 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg"] Oct 30 13:27:52.755261 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.755233 4931 cpu_manager.go:414] "containerMap: removing container" podUID="f314d798-3b21-4d5d-9f42-e1eee98c647c" containerName="istio-proxy" Oct 30 13:27:52.755261 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.755259 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f314d798-3b21-4d5d-9f42-e1eee98c647c" containerName="istio-proxy" Oct 30 13:27:52.755261 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.755302 4931 cpu_manager.go:414] "containerMap: removing container" podUID="7a109d1f-9e17-4638-a451-54d814f34fbd" containerName="maas-api" Oct 30 13:27:52.755556 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.755310 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a109d1f-9e17-4638-a451-54d814f34fbd" containerName="maas-api" Oct 30 13:27:52.755556 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.755329 4931 cpu_manager.go:414] "containerMap: removing container" podUID="914e1460-a166-4f84-8f6a-f84589bda525" containerName="revoke-keys" Oct 30 13:27:52.755556 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.755337 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="914e1460-a166-4f84-8f6a-f84589bda525" containerName="revoke-keys" Oct 30 13:27:52.755732 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.755645 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f314d798-3b21-4d5d-9f42-e1eee98c647c" containerName="istio-proxy" Oct 30 13:27:52.755732 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.755668 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f314d798-3b21-4d5d-9f42-e1eee98c647c" containerName="istio-proxy" Oct 30 13:27:52.755732 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.755688 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a109d1f-9e17-4638-a451-54d814f34fbd" containerName="maas-api" Oct 30 13:27:52.755732 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.755697 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="914e1460-a166-4f84-8f6a-f84589bda525" containerName="revoke-keys" Oct 30 13:27:52.760938 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.760912 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" Oct 30 13:27:52.792797 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.783439 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-bcd76c-openshift-default-dockercfg-znm5v\"" Oct 30 13:27:52.795941 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.795907 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg"] Oct 30 13:27:52.939088 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.939030 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6dbc7e-8785-4971-88c4-e3341faaadff-workload-socket\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" Oct 30 13:27:52.939312 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.939289 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c6dbc7e-8785-4971-88c4-e3341faaadff-istio-data\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" Oct 30 13:27:52.939451 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.939336 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6dbc7e-8785-4971-88c4-e3341faaadff-credential-socket\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" Oct 30 13:27:52.939451 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.939420 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c6dbc7e-8785-4971-88c4-e3341faaadff-istio-podinfo\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" Oct 30 13:27:52.939585 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.939533 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c6dbc7e-8785-4971-88c4-e3341faaadff-istio-envoy\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" Oct 30 13:27:52.939644 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.939632 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c6dbc7e-8785-4971-88c4-e3341faaadff-workload-certs\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" Oct 30 13:27:52.939704 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.939687 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c6dbc7e-8785-4971-88c4-e3341faaadff-istio-token\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" Oct 30 13:27:52.939767 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.939725 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c6dbc7e-8785-4971-88c4-e3341faaadff-istiod-ca-cert\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" Oct 30 13:27:52.939825 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:52.939808 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpz7s\" (UniqueName: \"kubernetes.io/projected/5c6dbc7e-8785-4971-88c4-e3341faaadff-kube-api-access-kpz7s\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" Oct 30 13:27:53.041034 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.040997 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpz7s\" (UniqueName: \"kubernetes.io/projected/5c6dbc7e-8785-4971-88c4-e3341faaadff-kube-api-access-kpz7s\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" Oct 30 13:27:53.041209 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.041062 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6dbc7e-8785-4971-88c4-e3341faaadff-workload-socket\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" Oct 30 13:27:53.041209 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.041151 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c6dbc7e-8785-4971-88c4-e3341faaadff-istio-data\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" Oct 30 13:27:53.041209 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.041183 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6dbc7e-8785-4971-88c4-e3341faaadff-credential-socket\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" Oct 30 13:27:53.041442 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.041241 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c6dbc7e-8785-4971-88c4-e3341faaadff-istio-podinfo\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" Oct 30 13:27:53.041442 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.041326 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c6dbc7e-8785-4971-88c4-e3341faaadff-istio-envoy\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" Oct 30 13:27:53.041442 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.041408 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c6dbc7e-8785-4971-88c4-e3341faaadff-workload-certs\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" Oct 30 13:27:53.041610 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.041447 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c6dbc7e-8785-4971-88c4-e3341faaadff-istio-token\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" Oct 30 13:27:53.041610 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.041484 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c6dbc7e-8785-4971-88c4-e3341faaadff-istiod-ca-cert\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" Oct 30 13:27:53.041610 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.041589 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c6dbc7e-8785-4971-88c4-e3341faaadff-istio-data\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" Oct 30 13:27:53.041772 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.041641 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6dbc7e-8785-4971-88c4-e3341faaadff-credential-socket\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" Oct 30 13:27:53.042153 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.041845 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c6dbc7e-8785-4971-88c4-e3341faaadff-workload-certs\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" Oct 30 13:27:53.042153 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.041957 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6dbc7e-8785-4971-88c4-e3341faaadff-workload-socket\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" Oct 30 13:27:53.042359 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.042201 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c6dbc7e-8785-4971-88c4-e3341faaadff-istiod-ca-cert\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" Oct 30 13:27:53.043817 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.043767 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c6dbc7e-8785-4971-88c4-e3341faaadff-istio-envoy\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" Oct 30 13:27:53.044420 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.044402 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c6dbc7e-8785-4971-88c4-e3341faaadff-istio-podinfo\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" Oct 30 13:27:53.066169 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.066146 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpz7s\" (UniqueName: \"kubernetes.io/projected/5c6dbc7e-8785-4971-88c4-e3341faaadff-kube-api-access-kpz7s\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" Oct 30 13:27:53.067398 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.067377 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c6dbc7e-8785-4971-88c4-e3341faaadff-istio-token\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" Oct 30 13:27:53.081385 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.081364 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" Oct 30 13:27:53.134230 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.134193 4931 generic.go:356] "Generic (PLEG): container finished" podID="f314d798-3b21-4d5d-9f42-e1eee98c647c" containerID="c5ee9c9e0b15271c1ad7397bbd81d05bb3843220349bf8028af678ccd3cf16e1" exitCode=0 Oct 30 13:27:53.134407 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.134261 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" event={"ID":"f314d798-3b21-4d5d-9f42-e1eee98c647c","Type":"ContainerDied","Data":"c5ee9c9e0b15271c1ad7397bbd81d05bb3843220349bf8028af678ccd3cf16e1"} Oct 30 13:27:53.134407 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.134313 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" Oct 30 13:27:53.134407 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.134366 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k" event={"ID":"f314d798-3b21-4d5d-9f42-e1eee98c647c","Type":"ContainerDied","Data":"7a7dc8025f22f9d6f24850d8326319b194950ccdaae5d5c4b799dfef98911660"} Oct 30 13:27:53.134407 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.134397 4931 scope.go:122] "RemoveContainer" containerID="c5ee9c9e0b15271c1ad7397bbd81d05bb3843220349bf8028af678ccd3cf16e1" Oct 30 13:27:53.156114 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.154747 4931 scope.go:122] "RemoveContainer" containerID="d161058b9dae5dcd307bb3376e0ce6a632b66d004dd38f2e575814367a69418e" Oct 30 13:27:53.176575 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.171858 4931 scope.go:122] "RemoveContainer" containerID="c5ee9c9e0b15271c1ad7397bbd81d05bb3843220349bf8028af678ccd3cf16e1" Oct 30 13:27:53.176575 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:27:53.172157 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5ee9c9e0b15271c1ad7397bbd81d05bb3843220349bf8028af678ccd3cf16e1\": container with ID starting with c5ee9c9e0b15271c1ad7397bbd81d05bb3843220349bf8028af678ccd3cf16e1 not found: ID does not exist" containerID="c5ee9c9e0b15271c1ad7397bbd81d05bb3843220349bf8028af678ccd3cf16e1" Oct 30 13:27:53.176575 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.172190 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5ee9c9e0b15271c1ad7397bbd81d05bb3843220349bf8028af678ccd3cf16e1"} err="failed to get container status \"c5ee9c9e0b15271c1ad7397bbd81d05bb3843220349bf8028af678ccd3cf16e1\": rpc error: code = NotFound desc = could not find container \"c5ee9c9e0b15271c1ad7397bbd81d05bb3843220349bf8028af678ccd3cf16e1\": container with ID starting with c5ee9c9e0b15271c1ad7397bbd81d05bb3843220349bf8028af678ccd3cf16e1 not found: ID does not exist" Oct 30 13:27:53.176575 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.172211 4931 scope.go:122] "RemoveContainer" containerID="d161058b9dae5dcd307bb3376e0ce6a632b66d004dd38f2e575814367a69418e" Oct 30 13:27:53.176575 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:27:53.172622 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d161058b9dae5dcd307bb3376e0ce6a632b66d004dd38f2e575814367a69418e\": container with ID starting with d161058b9dae5dcd307bb3376e0ce6a632b66d004dd38f2e575814367a69418e not found: ID does not exist" containerID="d161058b9dae5dcd307bb3376e0ce6a632b66d004dd38f2e575814367a69418e" Oct 30 13:27:53.176575 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.172655 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d161058b9dae5dcd307bb3376e0ce6a632b66d004dd38f2e575814367a69418e"} err="failed to get container status \"d161058b9dae5dcd307bb3376e0ce6a632b66d004dd38f2e575814367a69418e\": rpc error: code = NotFound desc = could not find container \"d161058b9dae5dcd307bb3376e0ce6a632b66d004dd38f2e575814367a69418e\": container with ID starting with d161058b9dae5dcd307bb3376e0ce6a632b66d004dd38f2e575814367a69418e not found: ID does not exist" Oct 30 13:27:53.200487 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.200461 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k"] Oct 30 13:27:53.213658 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.213631 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k"] Oct 30 13:27:53.241100 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.241068 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5e47c-8xp6x"] Oct 30 13:27:53.241901 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.241872 4931 cpu_manager.go:414] "containerMap: removing container" podUID="f314d798-3b21-4d5d-9f42-e1eee98c647c" containerName="istio-proxy" Oct 30 13:27:53.241960 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.241903 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f314d798-3b21-4d5d-9f42-e1eee98c647c" containerName="istio-proxy" Oct 30 13:27:53.247381 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.247359 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5e47c-8xp6x" Oct 30 13:27:53.264715 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.264600 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5e47c-8xp6x"] Oct 30 13:27:53.299422 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.299355 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:27:53.299528 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.299474 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:27:53.345181 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.344993 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89mg8\" (UniqueName: \"kubernetes.io/projected/aa455cb1-8f65-4233-85fb-d7e651d16140-kube-api-access-89mg8\") pod \"maas-api-revoke-keys-e2e-inf-b-a5e47c-8xp6x\" (UID: \"aa455cb1-8f65-4233-85fb-d7e651d16140\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5e47c-8xp6x" Oct 30 13:27:53.345355 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.345322 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa455cb1-8f65-4233-85fb-d7e651d16140-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a5e47c-8xp6x\" (UID: \"aa455cb1-8f65-4233-85fb-d7e651d16140\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5e47c-8xp6x" Oct 30 13:27:53.345427 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.345392 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa455cb1-8f65-4233-85fb-d7e651d16140-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a5e47c-8xp6x\" (UID: \"aa455cb1-8f65-4233-85fb-d7e651d16140\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5e47c-8xp6x" Oct 30 13:27:53.420184 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.419311 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg"] Oct 30 13:27:53.425218 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:27:53.425188 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c6dbc7e_8785_4971_88c4_e3341faaadff.slice/crio-7cd31c858742de0261dbdd8b7b2cee86600e83220a5fa3b474c154d202aa475a WatchSource:0}: Error finding container 7cd31c858742de0261dbdd8b7b2cee86600e83220a5fa3b474c154d202aa475a: Status 404 returned error can't find the container with id 7cd31c858742de0261dbdd8b7b2cee86600e83220a5fa3b474c154d202aa475a Oct 30 13:27:53.429243 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.429202 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:27:53.429391 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.429316 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:27:53.429391 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.429365 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:27:53.451962 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.446358 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa455cb1-8f65-4233-85fb-d7e651d16140-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a5e47c-8xp6x\" (UID: \"aa455cb1-8f65-4233-85fb-d7e651d16140\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5e47c-8xp6x" Oct 30 13:27:53.451962 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.446430 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa455cb1-8f65-4233-85fb-d7e651d16140-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a5e47c-8xp6x\" (UID: \"aa455cb1-8f65-4233-85fb-d7e651d16140\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5e47c-8xp6x" Oct 30 13:27:53.451962 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.446474 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89mg8\" (UniqueName: \"kubernetes.io/projected/aa455cb1-8f65-4233-85fb-d7e651d16140-kube-api-access-89mg8\") pod \"maas-api-revoke-keys-e2e-inf-b-a5e47c-8xp6x\" (UID: \"aa455cb1-8f65-4233-85fb-d7e651d16140\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5e47c-8xp6x" Oct 30 13:27:53.451962 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.449834 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa455cb1-8f65-4233-85fb-d7e651d16140-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a5e47c-8xp6x\" (UID: \"aa455cb1-8f65-4233-85fb-d7e651d16140\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5e47c-8xp6x" Oct 30 13:27:53.451962 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.451905 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa455cb1-8f65-4233-85fb-d7e651d16140-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a5e47c-8xp6x\" (UID: \"aa455cb1-8f65-4233-85fb-d7e651d16140\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5e47c-8xp6x" Oct 30 13:27:53.471261 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.471239 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89mg8\" (UniqueName: \"kubernetes.io/projected/aa455cb1-8f65-4233-85fb-d7e651d16140-kube-api-access-89mg8\") pod \"maas-api-revoke-keys-e2e-inf-b-a5e47c-8xp6x\" (UID: \"aa455cb1-8f65-4233-85fb-d7e651d16140\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5e47c-8xp6x" Oct 30 13:27:53.513850 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.513826 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f314d798-3b21-4d5d-9f42-e1eee98c647c" path="/var/lib/kubelet/pods/f314d798-3b21-4d5d-9f42-e1eee98c647c/volumes" Oct 30 13:27:53.561252 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.561196 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5e47c-8xp6x" Oct 30 13:27:53.896806 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:53.896774 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5e47c-8xp6x"] Oct 30 13:27:53.899720 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:27:53.899688 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa455cb1_8f65_4233_85fb_d7e651d16140.slice/crio-b1996a1b9179bde9970d6dbea440d9bfeff41434f55a1c2f7ca5744466f790d1 WatchSource:0}: Error finding container b1996a1b9179bde9970d6dbea440d9bfeff41434f55a1c2f7ca5744466f790d1: Status 404 returned error can't find the container with id b1996a1b9179bde9970d6dbea440d9bfeff41434f55a1c2f7ca5744466f790d1 Oct 30 13:27:54.143906 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:54.143866 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" event={"ID":"5c6dbc7e-8785-4971-88c4-e3341faaadff","Type":"ContainerStarted","Data":"9148f6cf302e45329dca322c30a067837c39207a4d11f48b7cb56acb3493f713"} Oct 30 13:27:54.144076 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:54.143936 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" event={"ID":"5c6dbc7e-8785-4971-88c4-e3341faaadff","Type":"ContainerStarted","Data":"7cd31c858742de0261dbdd8b7b2cee86600e83220a5fa3b474c154d202aa475a"} Oct 30 13:27:54.145332 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:54.145308 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5e47c-8xp6x" event={"ID":"aa455cb1-8f65-4233-85fb-d7e651d16140","Type":"ContainerStarted","Data":"b1996a1b9179bde9970d6dbea440d9bfeff41434f55a1c2f7ca5744466f790d1"} Oct 30 13:27:54.197702 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:54.197655 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" podStartSLOduration=2.197641994 podStartE2EDuration="2.197641994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:54.194200247 +0000 UTC m=+2572.797698695" watchObservedRunningTime="2026-10-30 13:27:54.197641994 +0000 UTC m=+2572.801140681" Oct 30 13:27:55.081919 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:55.081868 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" Oct 30 13:27:55.083735 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:55.083708 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 13:27:55.083863 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:55.083755 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" podUID="5c6dbc7e-8785-4971-88c4-e3341faaadff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 13:27:55.155467 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:55.155384 4931 generic.go:356] "Generic (PLEG): container finished" podID="aa455cb1-8f65-4233-85fb-d7e651d16140" containerID="883c0d16edc9c250e050ae9f56756f71bf6948b41eacadbc928690a967790464" exitCode=0 Oct 30 13:27:55.155617 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:55.155471 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5e47c-8xp6x" event={"ID":"aa455cb1-8f65-4233-85fb-d7e651d16140","Type":"ContainerDied","Data":"883c0d16edc9c250e050ae9f56756f71bf6948b41eacadbc928690a967790464"} Oct 30 13:27:56.081805 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:56.081765 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 13:27:56.081975 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:56.081827 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" podUID="5c6dbc7e-8785-4971-88c4-e3341faaadff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 13:27:56.367010 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:56.366944 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5e47c-8xp6x" Oct 30 13:27:56.386677 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:56.386647 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa455cb1-8f65-4233-85fb-d7e651d16140-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa455cb1-8f65-4233-85fb-d7e651d16140-maas-api-xks-ca\") pod \"aa455cb1-8f65-4233-85fb-d7e651d16140\" (UID: \"aa455cb1-8f65-4233-85fb-d7e651d16140\") " Oct 30 13:27:56.386836 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:56.386700 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa455cb1-8f65-4233-85fb-d7e651d16140-kube-api-access-89mg8\" (UniqueName: \"kubernetes.io/projected/aa455cb1-8f65-4233-85fb-d7e651d16140-kube-api-access-89mg8\") pod \"aa455cb1-8f65-4233-85fb-d7e651d16140\" (UID: \"aa455cb1-8f65-4233-85fb-d7e651d16140\") " Oct 30 13:27:56.386836 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:56.386746 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa455cb1-8f65-4233-85fb-d7e651d16140-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa455cb1-8f65-4233-85fb-d7e651d16140-maas-api-openshift-ca\") pod \"aa455cb1-8f65-4233-85fb-d7e651d16140\" (UID: \"aa455cb1-8f65-4233-85fb-d7e651d16140\") " Oct 30 13:27:56.387141 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:56.387108 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa455cb1-8f65-4233-85fb-d7e651d16140-maas-api-openshift-ca" pod "aa455cb1-8f65-4233-85fb-d7e651d16140" (UID: "aa455cb1-8f65-4233-85fb-d7e651d16140"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:27:56.387301 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:56.387256 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa455cb1-8f65-4233-85fb-d7e651d16140-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:27:56.388800 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:56.388776 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa455cb1-8f65-4233-85fb-d7e651d16140-maas-api-xks-ca" pod "aa455cb1-8f65-4233-85fb-d7e651d16140" (UID: "aa455cb1-8f65-4233-85fb-d7e651d16140"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:27:56.388966 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:56.388944 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa455cb1-8f65-4233-85fb-d7e651d16140-kube-api-access-89mg8" pod "aa455cb1-8f65-4233-85fb-d7e651d16140" (UID: "aa455cb1-8f65-4233-85fb-d7e651d16140"). InnerVolumeSpecName "kube-api-access-89mg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:56.488734 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:56.488700 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa455cb1-8f65-4233-85fb-d7e651d16140-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:27:56.488734 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:56.488732 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89mg8\" (UniqueName: \"kubernetes.io/projected/aa455cb1-8f65-4233-85fb-d7e651d16140-kube-api-access-89mg8\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:27:57.082086 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:57.081763 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 13:27:57.082086 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:57.081817 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" podUID="5c6dbc7e-8785-4971-88c4-e3341faaadff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 13:27:57.175481 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:57.175454 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5e47c-8xp6x" Oct 30 13:27:57.175674 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:57.175487 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5e47c-8xp6x" event={"ID":"aa455cb1-8f65-4233-85fb-d7e651d16140","Type":"ContainerDied","Data":"b1996a1b9179bde9970d6dbea440d9bfeff41434f55a1c2f7ca5744466f790d1"} Oct 30 13:27:57.175674 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:57.175535 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1996a1b9179bde9970d6dbea440d9bfeff41434f55a1c2f7ca5744466f790d1" Oct 30 13:27:58.081733 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:58.081695 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 13:27:58.081914 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:58.081770 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" podUID="5c6dbc7e-8785-4971-88c4-e3341faaadff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 13:27:59.082591 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:59.082554 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 13:27:59.082996 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:27:59.082611 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" podUID="5c6dbc7e-8785-4971-88c4-e3341faaadff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 13:28:00.081985 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:00.081942 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 13:28:00.082178 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:00.082003 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" podUID="5c6dbc7e-8785-4971-88c4-e3341faaadff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 13:28:01.082097 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:01.082052 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 13:28:01.082527 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:01.082124 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" podUID="5c6dbc7e-8785-4971-88c4-e3341faaadff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 13:28:02.089080 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:02.089047 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" Oct 30 13:28:02.089583 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:02.089500 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" Oct 30 13:28:02.090351 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:02.090327 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" Oct 30 13:28:05.183414 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:05.183373 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:28:07.407921 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:28:07.407883 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:28:10.534299 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:10.534252 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a132210-28hbg"] Oct 30 13:28:10.534817 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:10.534802 4931 cpu_manager.go:414] "containerMap: removing container" podUID="aa455cb1-8f65-4233-85fb-d7e651d16140" containerName="revoke-keys" Oct 30 13:28:10.534893 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:10.534820 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa455cb1-8f65-4233-85fb-d7e651d16140" containerName="revoke-keys" Oct 30 13:28:10.534937 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:10.534899 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa455cb1-8f65-4233-85fb-d7e651d16140" containerName="revoke-keys" Oct 30 13:28:10.539777 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:10.539754 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a132210-28hbg" Oct 30 13:28:10.568926 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:10.568897 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a132210-28hbg"] Oct 30 13:28:10.577145 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:10.577118 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:28:10.577297 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:10.577169 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:28:10.627913 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:10.627877 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm65t\" (UniqueName: \"kubernetes.io/projected/d62dd022-0695-4a89-8f8c-04d00f5dffed-kube-api-access-cm65t\") pod \"maas-api-revoke-sub-7a132210-28hbg\" (UID: \"d62dd022-0695-4a89-8f8c-04d00f5dffed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a132210-28hbg" Oct 30 13:28:10.628079 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:10.628030 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d62dd022-0695-4a89-8f8c-04d00f5dffed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a132210-28hbg\" (UID: \"d62dd022-0695-4a89-8f8c-04d00f5dffed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a132210-28hbg" Oct 30 13:28:10.628159 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:10.628145 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d62dd022-0695-4a89-8f8c-04d00f5dffed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a132210-28hbg\" (UID: \"d62dd022-0695-4a89-8f8c-04d00f5dffed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a132210-28hbg" Oct 30 13:28:10.729295 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:10.729249 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d62dd022-0695-4a89-8f8c-04d00f5dffed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a132210-28hbg\" (UID: \"d62dd022-0695-4a89-8f8c-04d00f5dffed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a132210-28hbg" Oct 30 13:28:10.729504 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:10.729351 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm65t\" (UniqueName: \"kubernetes.io/projected/d62dd022-0695-4a89-8f8c-04d00f5dffed-kube-api-access-cm65t\") pod \"maas-api-revoke-sub-7a132210-28hbg\" (UID: \"d62dd022-0695-4a89-8f8c-04d00f5dffed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a132210-28hbg" Oct 30 13:28:10.729504 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:10.729461 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d62dd022-0695-4a89-8f8c-04d00f5dffed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a132210-28hbg\" (UID: \"d62dd022-0695-4a89-8f8c-04d00f5dffed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a132210-28hbg" Oct 30 13:28:10.730109 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:10.730087 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d62dd022-0695-4a89-8f8c-04d00f5dffed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a132210-28hbg\" (UID: \"d62dd022-0695-4a89-8f8c-04d00f5dffed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a132210-28hbg" Oct 30 13:28:10.731683 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:10.731662 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d62dd022-0695-4a89-8f8c-04d00f5dffed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a132210-28hbg\" (UID: \"d62dd022-0695-4a89-8f8c-04d00f5dffed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a132210-28hbg" Oct 30 13:28:10.754306 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:10.754250 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm65t\" (UniqueName: \"kubernetes.io/projected/d62dd022-0695-4a89-8f8c-04d00f5dffed-kube-api-access-cm65t\") pod \"maas-api-revoke-sub-7a132210-28hbg\" (UID: \"d62dd022-0695-4a89-8f8c-04d00f5dffed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a132210-28hbg" Oct 30 13:28:10.854249 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:10.854176 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a132210-28hbg" Oct 30 13:28:11.163905 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:11.163865 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a132210-28hbg"] Oct 30 13:28:11.164504 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:28:11.164478 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd62dd022_0695_4a89_8f8c_04d00f5dffed.slice/crio-93b5549526a6b3605294b72154b5946347ee7cc384b27245924d2a5921fdac35 WatchSource:0}: Error finding container 93b5549526a6b3605294b72154b5946347ee7cc384b27245924d2a5921fdac35: Status 404 returned error can't find the container with id 93b5549526a6b3605294b72154b5946347ee7cc384b27245924d2a5921fdac35 Oct 30 13:28:11.321362 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:11.321320 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a132210-28hbg" event={"ID":"d62dd022-0695-4a89-8f8c-04d00f5dffed","Type":"ContainerStarted","Data":"93b5549526a6b3605294b72154b5946347ee7cc384b27245924d2a5921fdac35"} Oct 30 13:28:11.798348 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:11.798257 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d67fc6-tq2z2"] Oct 30 13:28:11.802412 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:11.802393 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d67fc6-tq2z2" Oct 30 13:28:11.825724 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:11.825696 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d67fc6-tq2z2"] Oct 30 13:28:11.842666 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:11.842640 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a612b14-5b32-4fc1-a656-2e2fab10530a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59d67fc6-tq2z2\" (UID: \"7a612b14-5b32-4fc1-a656-2e2fab10530a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d67fc6-tq2z2" Oct 30 13:28:11.842811 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:11.842777 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a612b14-5b32-4fc1-a656-2e2fab10530a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59d67fc6-tq2z2\" (UID: \"7a612b14-5b32-4fc1-a656-2e2fab10530a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d67fc6-tq2z2" Oct 30 13:28:11.842895 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:11.842823 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vklcv\" (UniqueName: \"kubernetes.io/projected/7a612b14-5b32-4fc1-a656-2e2fab10530a-kube-api-access-vklcv\") pod \"maas-api-revoke-sub-59d67fc6-tq2z2\" (UID: \"7a612b14-5b32-4fc1-a656-2e2fab10530a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d67fc6-tq2z2" Oct 30 13:28:11.944253 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:11.944219 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a612b14-5b32-4fc1-a656-2e2fab10530a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59d67fc6-tq2z2\" (UID: \"7a612b14-5b32-4fc1-a656-2e2fab10530a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d67fc6-tq2z2" Oct 30 13:28:11.944453 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:11.944338 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a612b14-5b32-4fc1-a656-2e2fab10530a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59d67fc6-tq2z2\" (UID: \"7a612b14-5b32-4fc1-a656-2e2fab10530a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d67fc6-tq2z2" Oct 30 13:28:11.944453 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:11.944381 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vklcv\" (UniqueName: \"kubernetes.io/projected/7a612b14-5b32-4fc1-a656-2e2fab10530a-kube-api-access-vklcv\") pod \"maas-api-revoke-sub-59d67fc6-tq2z2\" (UID: \"7a612b14-5b32-4fc1-a656-2e2fab10530a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d67fc6-tq2z2" Oct 30 13:28:11.944985 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:11.944957 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a612b14-5b32-4fc1-a656-2e2fab10530a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59d67fc6-tq2z2\" (UID: \"7a612b14-5b32-4fc1-a656-2e2fab10530a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d67fc6-tq2z2" Oct 30 13:28:11.946539 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:11.946518 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a612b14-5b32-4fc1-a656-2e2fab10530a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59d67fc6-tq2z2\" (UID: \"7a612b14-5b32-4fc1-a656-2e2fab10530a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d67fc6-tq2z2" Oct 30 13:28:11.994662 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:11.994583 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vklcv\" (UniqueName: \"kubernetes.io/projected/7a612b14-5b32-4fc1-a656-2e2fab10530a-kube-api-access-vklcv\") pod \"maas-api-revoke-sub-59d67fc6-tq2z2\" (UID: \"7a612b14-5b32-4fc1-a656-2e2fab10530a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d67fc6-tq2z2" Oct 30 13:28:12.117210 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:12.117145 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d67fc6-tq2z2" Oct 30 13:28:12.329361 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:12.329325 4931 generic.go:356] "Generic (PLEG): container finished" podID="d62dd022-0695-4a89-8f8c-04d00f5dffed" containerID="60b23916487596c5730fc594672c71185b3755a7f8ebe49ce6ec673a4ac92723" exitCode=0 Oct 30 13:28:12.329529 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:12.329413 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a132210-28hbg" event={"ID":"d62dd022-0695-4a89-8f8c-04d00f5dffed","Type":"ContainerDied","Data":"60b23916487596c5730fc594672c71185b3755a7f8ebe49ce6ec673a4ac92723"} Oct 30 13:28:12.424432 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:12.424402 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d67fc6-tq2z2"] Oct 30 13:28:12.426070 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:28:12.426042 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a612b14_5b32_4fc1_a656_2e2fab10530a.slice/crio-d594b16201eacfce6aaccaa66b777738acfc385e1c6c788082371472eab9e40c WatchSource:0}: Error finding container d594b16201eacfce6aaccaa66b777738acfc385e1c6c788082371472eab9e40c: Status 404 returned error can't find the container with id d594b16201eacfce6aaccaa66b777738acfc385e1c6c788082371472eab9e40c Oct 30 13:28:13.336910 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:13.336877 4931 generic.go:356] "Generic (PLEG): container finished" podID="7a612b14-5b32-4fc1-a656-2e2fab10530a" containerID="625f65074c752fb757e27289d4a460b75cfebba9149f5d8e9801726a7ce83de6" exitCode=0 Oct 30 13:28:13.337254 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:13.336958 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d67fc6-tq2z2" event={"ID":"7a612b14-5b32-4fc1-a656-2e2fab10530a","Type":"ContainerDied","Data":"625f65074c752fb757e27289d4a460b75cfebba9149f5d8e9801726a7ce83de6"} Oct 30 13:28:13.337254 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:13.337015 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d67fc6-tq2z2" event={"ID":"7a612b14-5b32-4fc1-a656-2e2fab10530a","Type":"ContainerStarted","Data":"d594b16201eacfce6aaccaa66b777738acfc385e1c6c788082371472eab9e40c"} Oct 30 13:28:13.499056 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:13.499031 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a132210-28hbg" Oct 30 13:28:13.563788 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:13.563706 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d62dd022-0695-4a89-8f8c-04d00f5dffed-kube-api-access-cm65t\" (UniqueName: \"kubernetes.io/projected/d62dd022-0695-4a89-8f8c-04d00f5dffed-kube-api-access-cm65t\") pod \"d62dd022-0695-4a89-8f8c-04d00f5dffed\" (UID: \"d62dd022-0695-4a89-8f8c-04d00f5dffed\") " Oct 30 13:28:13.563788 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:13.563761 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d62dd022-0695-4a89-8f8c-04d00f5dffed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d62dd022-0695-4a89-8f8c-04d00f5dffed-maas-api-openshift-ca\") pod \"d62dd022-0695-4a89-8f8c-04d00f5dffed\" (UID: \"d62dd022-0695-4a89-8f8c-04d00f5dffed\") " Oct 30 13:28:13.563788 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:13.563784 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d62dd022-0695-4a89-8f8c-04d00f5dffed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d62dd022-0695-4a89-8f8c-04d00f5dffed-maas-api-xks-ca\") pod \"d62dd022-0695-4a89-8f8c-04d00f5dffed\" (UID: \"d62dd022-0695-4a89-8f8c-04d00f5dffed\") " Oct 30 13:28:13.564213 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:13.564183 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d62dd022-0695-4a89-8f8c-04d00f5dffed-maas-api-openshift-ca" pod "d62dd022-0695-4a89-8f8c-04d00f5dffed" (UID: "d62dd022-0695-4a89-8f8c-04d00f5dffed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:28:13.564425 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:13.564403 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d62dd022-0695-4a89-8f8c-04d00f5dffed-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:28:13.566003 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:13.565976 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d62dd022-0695-4a89-8f8c-04d00f5dffed-kube-api-access-cm65t" pod "d62dd022-0695-4a89-8f8c-04d00f5dffed" (UID: "d62dd022-0695-4a89-8f8c-04d00f5dffed"). InnerVolumeSpecName "kube-api-access-cm65t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:28:13.566203 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:13.566182 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d62dd022-0695-4a89-8f8c-04d00f5dffed-maas-api-xks-ca" pod "d62dd022-0695-4a89-8f8c-04d00f5dffed" (UID: "d62dd022-0695-4a89-8f8c-04d00f5dffed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:28:13.666116 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:13.666083 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm65t\" (UniqueName: \"kubernetes.io/projected/d62dd022-0695-4a89-8f8c-04d00f5dffed-kube-api-access-cm65t\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:28:13.666116 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:13.666115 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d62dd022-0695-4a89-8f8c-04d00f5dffed-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:28:14.344777 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:14.344745 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a132210-28hbg" Oct 30 13:28:14.345201 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:14.344781 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a132210-28hbg" event={"ID":"d62dd022-0695-4a89-8f8c-04d00f5dffed","Type":"ContainerDied","Data":"93b5549526a6b3605294b72154b5946347ee7cc384b27245924d2a5921fdac35"} Oct 30 13:28:14.345201 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:14.344838 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93b5549526a6b3605294b72154b5946347ee7cc384b27245924d2a5921fdac35" Oct 30 13:28:14.505419 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:14.505395 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d67fc6-tq2z2" Oct 30 13:28:14.582060 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:14.582026 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a612b14-5b32-4fc1-a656-2e2fab10530a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a612b14-5b32-4fc1-a656-2e2fab10530a-maas-api-xks-ca\") pod \"7a612b14-5b32-4fc1-a656-2e2fab10530a\" (UID: \"7a612b14-5b32-4fc1-a656-2e2fab10530a\") " Oct 30 13:28:14.582252 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:14.582150 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a612b14-5b32-4fc1-a656-2e2fab10530a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a612b14-5b32-4fc1-a656-2e2fab10530a-maas-api-openshift-ca\") pod \"7a612b14-5b32-4fc1-a656-2e2fab10530a\" (UID: \"7a612b14-5b32-4fc1-a656-2e2fab10530a\") " Oct 30 13:28:14.582389 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:14.582374 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a612b14-5b32-4fc1-a656-2e2fab10530a-kube-api-access-vklcv\" (UniqueName: \"kubernetes.io/projected/7a612b14-5b32-4fc1-a656-2e2fab10530a-kube-api-access-vklcv\") pod \"7a612b14-5b32-4fc1-a656-2e2fab10530a\" (UID: \"7a612b14-5b32-4fc1-a656-2e2fab10530a\") " Oct 30 13:28:14.582610 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:14.582572 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a612b14-5b32-4fc1-a656-2e2fab10530a-maas-api-openshift-ca" pod "7a612b14-5b32-4fc1-a656-2e2fab10530a" (UID: "7a612b14-5b32-4fc1-a656-2e2fab10530a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:28:14.582952 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:14.582928 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a612b14-5b32-4fc1-a656-2e2fab10530a-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:28:14.584396 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:14.584370 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a612b14-5b32-4fc1-a656-2e2fab10530a-maas-api-xks-ca" pod "7a612b14-5b32-4fc1-a656-2e2fab10530a" (UID: "7a612b14-5b32-4fc1-a656-2e2fab10530a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:28:14.584477 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:14.584414 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a612b14-5b32-4fc1-a656-2e2fab10530a-kube-api-access-vklcv" pod "7a612b14-5b32-4fc1-a656-2e2fab10530a" (UID: "7a612b14-5b32-4fc1-a656-2e2fab10530a"). InnerVolumeSpecName "kube-api-access-vklcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:28:14.684484 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:14.684452 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vklcv\" (UniqueName: \"kubernetes.io/projected/7a612b14-5b32-4fc1-a656-2e2fab10530a-kube-api-access-vklcv\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:28:14.684484 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:14.684479 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a612b14-5b32-4fc1-a656-2e2fab10530a-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:28:15.353315 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:15.353286 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d67fc6-tq2z2" Oct 30 13:28:15.353713 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:15.353318 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d67fc6-tq2z2" event={"ID":"7a612b14-5b32-4fc1-a656-2e2fab10530a","Type":"ContainerDied","Data":"d594b16201eacfce6aaccaa66b777738acfc385e1c6c788082371472eab9e40c"} Oct 30 13:28:15.353713 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:15.353371 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d594b16201eacfce6aaccaa66b777738acfc385e1c6c788082371472eab9e40c" Oct 30 13:28:20.981999 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:20.981960 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbbe51fe-9b99z"] Oct 30 13:28:20.982574 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:20.982558 4931 cpu_manager.go:414] "containerMap: removing container" podUID="7a612b14-5b32-4fc1-a656-2e2fab10530a" containerName="revoke-keys" Oct 30 13:28:20.982622 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:20.982576 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a612b14-5b32-4fc1-a656-2e2fab10530a" containerName="revoke-keys" Oct 30 13:28:20.982663 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:20.982637 4931 cpu_manager.go:414] "containerMap: removing container" podUID="d62dd022-0695-4a89-8f8c-04d00f5dffed" containerName="revoke-keys" Oct 30 13:28:20.982663 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:20.982642 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d62dd022-0695-4a89-8f8c-04d00f5dffed" containerName="revoke-keys" Oct 30 13:28:20.982736 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:20.982727 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="d62dd022-0695-4a89-8f8c-04d00f5dffed" containerName="revoke-keys" Oct 30 13:28:20.982769 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:20.982739 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a612b14-5b32-4fc1-a656-2e2fab10530a" containerName="revoke-keys" Oct 30 13:28:20.986632 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:20.986610 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbbe51fe-9b99z" Oct 30 13:28:21.009773 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:21.009751 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:28:21.015193 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:21.015169 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbbe51fe-9b99z"] Oct 30 13:28:21.017904 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:21.017874 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:28:21.070102 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:21.070062 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87e0e82a-63e8-4aca-bfb5-80ac6a235c5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbbe51fe-9b99z\" (UID: \"87e0e82a-63e8-4aca-bfb5-80ac6a235c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbbe51fe-9b99z" Oct 30 13:28:21.070288 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:21.070121 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmwpr\" (UniqueName: \"kubernetes.io/projected/87e0e82a-63e8-4aca-bfb5-80ac6a235c5d-kube-api-access-xmwpr\") pod \"maas-api-revoke-sub-fbbe51fe-9b99z\" (UID: \"87e0e82a-63e8-4aca-bfb5-80ac6a235c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbbe51fe-9b99z" Oct 30 13:28:21.070366 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:21.070342 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87e0e82a-63e8-4aca-bfb5-80ac6a235c5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbbe51fe-9b99z\" (UID: \"87e0e82a-63e8-4aca-bfb5-80ac6a235c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbbe51fe-9b99z" Oct 30 13:28:21.171496 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:21.171467 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87e0e82a-63e8-4aca-bfb5-80ac6a235c5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbbe51fe-9b99z\" (UID: \"87e0e82a-63e8-4aca-bfb5-80ac6a235c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbbe51fe-9b99z" Oct 30 13:28:21.171661 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:21.171631 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87e0e82a-63e8-4aca-bfb5-80ac6a235c5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbbe51fe-9b99z\" (UID: \"87e0e82a-63e8-4aca-bfb5-80ac6a235c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbbe51fe-9b99z" Oct 30 13:28:21.171732 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:21.171712 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmwpr\" (UniqueName: \"kubernetes.io/projected/87e0e82a-63e8-4aca-bfb5-80ac6a235c5d-kube-api-access-xmwpr\") pod \"maas-api-revoke-sub-fbbe51fe-9b99z\" (UID: \"87e0e82a-63e8-4aca-bfb5-80ac6a235c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbbe51fe-9b99z" Oct 30 13:28:21.172170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:21.172147 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87e0e82a-63e8-4aca-bfb5-80ac6a235c5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbbe51fe-9b99z\" (UID: \"87e0e82a-63e8-4aca-bfb5-80ac6a235c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbbe51fe-9b99z" Oct 30 13:28:21.174398 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:21.174377 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87e0e82a-63e8-4aca-bfb5-80ac6a235c5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbbe51fe-9b99z\" (UID: \"87e0e82a-63e8-4aca-bfb5-80ac6a235c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbbe51fe-9b99z" Oct 30 13:28:21.197517 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:21.197492 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmwpr\" (UniqueName: \"kubernetes.io/projected/87e0e82a-63e8-4aca-bfb5-80ac6a235c5d-kube-api-access-xmwpr\") pod \"maas-api-revoke-sub-fbbe51fe-9b99z\" (UID: \"87e0e82a-63e8-4aca-bfb5-80ac6a235c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbbe51fe-9b99z" Oct 30 13:28:21.300481 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:21.300404 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbbe51fe-9b99z" Oct 30 13:28:21.631352 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:21.627325 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbbe51fe-9b99z"] Oct 30 13:28:22.272140 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:22.272062 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1155b409-rwgjs"] Oct 30 13:28:22.275995 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:22.275973 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1155b409-rwgjs" Oct 30 13:28:22.303467 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:22.303435 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1155b409-rwgjs"] Oct 30 13:28:22.384846 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:22.384806 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfc8h\" (UniqueName: \"kubernetes.io/projected/20d5c662-db5a-49ac-acb7-4f8c094297fd-kube-api-access-zfc8h\") pod \"maas-api-revoke-sub-1155b409-rwgjs\" (UID: \"20d5c662-db5a-49ac-acb7-4f8c094297fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1155b409-rwgjs" Oct 30 13:28:22.385029 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:22.384879 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d5c662-db5a-49ac-acb7-4f8c094297fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1155b409-rwgjs\" (UID: \"20d5c662-db5a-49ac-acb7-4f8c094297fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1155b409-rwgjs" Oct 30 13:28:22.385029 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:22.384931 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d5c662-db5a-49ac-acb7-4f8c094297fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1155b409-rwgjs\" (UID: \"20d5c662-db5a-49ac-acb7-4f8c094297fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1155b409-rwgjs" Oct 30 13:28:22.417541 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:22.417503 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbbe51fe-9b99z" event={"ID":"87e0e82a-63e8-4aca-bfb5-80ac6a235c5d","Type":"ContainerStarted","Data":"d1cb79df2f61e26246f0d70a95b39f3ce7c8a663c401ad801480f6fd53ce8f33"} Oct 30 13:28:22.417735 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:22.417559 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbbe51fe-9b99z" event={"ID":"87e0e82a-63e8-4aca-bfb5-80ac6a235c5d","Type":"ContainerStarted","Data":"cc758dc374769b0fde78f45707cfcd169b3c58801e08078e31cde4ee437c8a34"} Oct 30 13:28:22.470840 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:22.470784 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbbe51fe-9b99z" podStartSLOduration=2.470769768 podStartE2EDuration="2.470769768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:22.466862809 +0000 UTC m=+2601.070361255" watchObservedRunningTime="2026-10-30 13:28:22.470769768 +0000 UTC m=+2601.074268214" Oct 30 13:28:22.485939 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:22.485907 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfc8h\" (UniqueName: \"kubernetes.io/projected/20d5c662-db5a-49ac-acb7-4f8c094297fd-kube-api-access-zfc8h\") pod \"maas-api-revoke-sub-1155b409-rwgjs\" (UID: \"20d5c662-db5a-49ac-acb7-4f8c094297fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1155b409-rwgjs" Oct 30 13:28:22.486115 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:22.486089 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d5c662-db5a-49ac-acb7-4f8c094297fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1155b409-rwgjs\" (UID: \"20d5c662-db5a-49ac-acb7-4f8c094297fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1155b409-rwgjs" Oct 30 13:28:22.486192 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:22.486127 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d5c662-db5a-49ac-acb7-4f8c094297fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1155b409-rwgjs\" (UID: \"20d5c662-db5a-49ac-acb7-4f8c094297fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1155b409-rwgjs" Oct 30 13:28:22.486748 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:22.486726 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d5c662-db5a-49ac-acb7-4f8c094297fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1155b409-rwgjs\" (UID: \"20d5c662-db5a-49ac-acb7-4f8c094297fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1155b409-rwgjs" Oct 30 13:28:22.488749 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:22.488727 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d5c662-db5a-49ac-acb7-4f8c094297fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1155b409-rwgjs\" (UID: \"20d5c662-db5a-49ac-acb7-4f8c094297fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1155b409-rwgjs" Oct 30 13:28:22.510617 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:22.510593 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfc8h\" (UniqueName: \"kubernetes.io/projected/20d5c662-db5a-49ac-acb7-4f8c094297fd-kube-api-access-zfc8h\") pod \"maas-api-revoke-sub-1155b409-rwgjs\" (UID: \"20d5c662-db5a-49ac-acb7-4f8c094297fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1155b409-rwgjs" Oct 30 13:28:22.591158 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:22.591127 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1155b409-rwgjs" Oct 30 13:28:22.878135 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:22.878053 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg"] Oct 30 13:28:22.888322 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:22.883933 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" Oct 30 13:28:22.930247 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:22.930214 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-e9d641-openshift-default-dockercfg-jhrsg\"" Oct 30 13:28:22.931919 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:28:22.931725 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d5c662_db5a_49ac_acb7_4f8c094297fd.slice/crio-400c0feed6499659811b8f2aeadf6779ee34045e728e52b59e1afb19e1c2f7d3 WatchSource:0}: Error finding container 400c0feed6499659811b8f2aeadf6779ee34045e728e52b59e1afb19e1c2f7d3: Status 404 returned error can't find the container with id 400c0feed6499659811b8f2aeadf6779ee34045e728e52b59e1afb19e1c2f7d3 Oct 30 13:28:22.950315 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:22.949994 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1155b409-rwgjs"] Oct 30 13:28:22.955910 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:22.955879 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg"] Oct 30 13:28:22.993086 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:22.993056 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ba44c24-d5d3-461a-b2b3-bca6e225185c-istio-data\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" Oct 30 13:28:22.993216 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:22.993099 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ba44c24-d5d3-461a-b2b3-bca6e225185c-istiod-ca-cert\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" Oct 30 13:28:22.993216 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:22.993185 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ba44c24-d5d3-461a-b2b3-bca6e225185c-istio-token\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" Oct 30 13:28:22.993358 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:22.993250 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba44c24-d5d3-461a-b2b3-bca6e225185c-workload-certs\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" Oct 30 13:28:22.993411 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:22.993396 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blnwg\" (UniqueName: \"kubernetes.io/projected/6ba44c24-d5d3-461a-b2b3-bca6e225185c-kube-api-access-blnwg\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" Oct 30 13:28:22.993451 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:22.993419 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ba44c24-d5d3-461a-b2b3-bca6e225185c-istio-podinfo\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" Oct 30 13:28:22.993451 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:22.993443 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ba44c24-d5d3-461a-b2b3-bca6e225185c-istio-envoy\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" Oct 30 13:28:22.993545 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:22.993530 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba44c24-d5d3-461a-b2b3-bca6e225185c-credential-socket\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" Oct 30 13:28:22.993637 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:22.993616 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba44c24-d5d3-461a-b2b3-bca6e225185c-workload-socket\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" Oct 30 13:28:23.094934 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:23.094902 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ba44c24-d5d3-461a-b2b3-bca6e225185c-istio-data\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" Oct 30 13:28:23.095104 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:23.094940 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ba44c24-d5d3-461a-b2b3-bca6e225185c-istiod-ca-cert\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" Oct 30 13:28:23.095104 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:23.095069 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ba44c24-d5d3-461a-b2b3-bca6e225185c-istio-token\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" Oct 30 13:28:23.095232 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:23.095143 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba44c24-d5d3-461a-b2b3-bca6e225185c-workload-certs\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" Oct 30 13:28:23.095329 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:23.095230 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blnwg\" (UniqueName: \"kubernetes.io/projected/6ba44c24-d5d3-461a-b2b3-bca6e225185c-kube-api-access-blnwg\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" Oct 30 13:28:23.095329 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:23.095257 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ba44c24-d5d3-461a-b2b3-bca6e225185c-istio-podinfo\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" Oct 30 13:28:23.095329 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:23.095308 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ba44c24-d5d3-461a-b2b3-bca6e225185c-istio-envoy\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" Oct 30 13:28:23.095497 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:23.095342 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ba44c24-d5d3-461a-b2b3-bca6e225185c-istio-data\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" Oct 30 13:28:23.095497 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:23.095356 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba44c24-d5d3-461a-b2b3-bca6e225185c-credential-socket\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" Oct 30 13:28:23.095497 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:23.095489 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba44c24-d5d3-461a-b2b3-bca6e225185c-workload-socket\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" Oct 30 13:28:23.095716 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:23.095560 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba44c24-d5d3-461a-b2b3-bca6e225185c-workload-certs\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" Oct 30 13:28:23.095716 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:23.095659 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba44c24-d5d3-461a-b2b3-bca6e225185c-credential-socket\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" Oct 30 13:28:23.095716 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:23.095696 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ba44c24-d5d3-461a-b2b3-bca6e225185c-istiod-ca-cert\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" Oct 30 13:28:23.095880 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:23.095838 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba44c24-d5d3-461a-b2b3-bca6e225185c-workload-socket\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" Oct 30 13:28:23.097602 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:23.097581 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ba44c24-d5d3-461a-b2b3-bca6e225185c-istio-envoy\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" Oct 30 13:28:23.097902 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:23.097876 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ba44c24-d5d3-461a-b2b3-bca6e225185c-istio-podinfo\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" Oct 30 13:28:23.119258 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:23.119228 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ba44c24-d5d3-461a-b2b3-bca6e225185c-istio-token\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" Oct 30 13:28:23.119441 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:23.119423 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blnwg\" (UniqueName: \"kubernetes.io/projected/6ba44c24-d5d3-461a-b2b3-bca6e225185c-kube-api-access-blnwg\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" Oct 30 13:28:23.238905 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:23.238876 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" Oct 30 13:28:23.433066 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:23.432963 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1155b409-rwgjs" event={"ID":"20d5c662-db5a-49ac-acb7-4f8c094297fd","Type":"ContainerStarted","Data":"fb543fd7dae3a86556c1747536d0658860722d2b99dcebbac77ec65d691b2434"} Oct 30 13:28:23.433066 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:23.433034 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1155b409-rwgjs" event={"ID":"20d5c662-db5a-49ac-acb7-4f8c094297fd","Type":"ContainerStarted","Data":"400c0feed6499659811b8f2aeadf6779ee34045e728e52b59e1afb19e1c2f7d3"} Oct 30 13:28:23.435682 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:23.435651 4931 generic.go:356] "Generic (PLEG): container finished" podID="87e0e82a-63e8-4aca-bfb5-80ac6a235c5d" containerID="d1cb79df2f61e26246f0d70a95b39f3ce7c8a663c401ad801480f6fd53ce8f33" exitCode=0 Oct 30 13:28:23.435804 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:23.435762 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbbe51fe-9b99z" event={"ID":"87e0e82a-63e8-4aca-bfb5-80ac6a235c5d","Type":"ContainerDied","Data":"d1cb79df2f61e26246f0d70a95b39f3ce7c8a663c401ad801480f6fd53ce8f33"} Oct 30 13:28:23.487505 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:23.487234 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1155b409-rwgjs" podStartSLOduration=1.487189077 podStartE2EDuration="1.487189077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:23.475463529 +0000 UTC m=+2602.078961973" watchObservedRunningTime="2026-10-30 13:28:23.487189077 +0000 UTC m=+2602.090687524" Oct 30 13:28:23.576498 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:23.576466 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg"] Oct 30 13:28:23.577522 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:28:23.577496 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ba44c24_d5d3_461a_b2b3_bca6e225185c.slice/crio-3169f4914961b5da9d01ba12ccfd59be05c0f92376e8905ef8107e9f24cfe0fb WatchSource:0}: Error finding container 3169f4914961b5da9d01ba12ccfd59be05c0f92376e8905ef8107e9f24cfe0fb: Status 404 returned error can't find the container with id 3169f4914961b5da9d01ba12ccfd59be05c0f92376e8905ef8107e9f24cfe0fb Oct 30 13:28:23.580602 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:23.580570 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:28:23.580685 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:23.580632 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:28:23.580685 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:23.580661 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:28:24.458486 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:24.458444 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" event={"ID":"6ba44c24-d5d3-461a-b2b3-bca6e225185c","Type":"ContainerStarted","Data":"d05dd6d57a2a681c908fc5e3e9b045013d307d852b1ba7c250b09ebe78f615cf"} Oct 30 13:28:24.458979 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:24.458516 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" event={"ID":"6ba44c24-d5d3-461a-b2b3-bca6e225185c","Type":"ContainerStarted","Data":"3169f4914961b5da9d01ba12ccfd59be05c0f92376e8905ef8107e9f24cfe0fb"} Oct 30 13:28:24.462007 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:24.461978 4931 generic.go:356] "Generic (PLEG): container finished" podID="20d5c662-db5a-49ac-acb7-4f8c094297fd" containerID="fb543fd7dae3a86556c1747536d0658860722d2b99dcebbac77ec65d691b2434" exitCode=0 Oct 30 13:28:24.462543 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:24.462516 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1155b409-rwgjs" event={"ID":"20d5c662-db5a-49ac-acb7-4f8c094297fd","Type":"ContainerDied","Data":"fb543fd7dae3a86556c1747536d0658860722d2b99dcebbac77ec65d691b2434"} Oct 30 13:28:24.530550 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:24.530344 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" podStartSLOduration=2.530326977 podStartE2EDuration="2.530326977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:24.524880954 +0000 UTC m=+2603.128379399" watchObservedRunningTime="2026-10-30 13:28:24.530326977 +0000 UTC m=+2603.133825422" Oct 30 13:28:24.678066 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:24.678034 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbbe51fe-9b99z" Oct 30 13:28:24.823940 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:24.823853 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87e0e82a-63e8-4aca-bfb5-80ac6a235c5d-kube-api-access-xmwpr\" (UniqueName: \"kubernetes.io/projected/87e0e82a-63e8-4aca-bfb5-80ac6a235c5d-kube-api-access-xmwpr\") pod \"87e0e82a-63e8-4aca-bfb5-80ac6a235c5d\" (UID: \"87e0e82a-63e8-4aca-bfb5-80ac6a235c5d\") " Oct 30 13:28:24.824124 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:24.823967 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87e0e82a-63e8-4aca-bfb5-80ac6a235c5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87e0e82a-63e8-4aca-bfb5-80ac6a235c5d-maas-api-openshift-ca\") pod \"87e0e82a-63e8-4aca-bfb5-80ac6a235c5d\" (UID: \"87e0e82a-63e8-4aca-bfb5-80ac6a235c5d\") " Oct 30 13:28:24.824124 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:24.824031 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87e0e82a-63e8-4aca-bfb5-80ac6a235c5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87e0e82a-63e8-4aca-bfb5-80ac6a235c5d-maas-api-xks-ca\") pod \"87e0e82a-63e8-4aca-bfb5-80ac6a235c5d\" (UID: \"87e0e82a-63e8-4aca-bfb5-80ac6a235c5d\") " Oct 30 13:28:24.824409 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:24.824382 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e0e82a-63e8-4aca-bfb5-80ac6a235c5d-maas-api-openshift-ca" pod "87e0e82a-63e8-4aca-bfb5-80ac6a235c5d" (UID: "87e0e82a-63e8-4aca-bfb5-80ac6a235c5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:28:24.824540 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:24.824525 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87e0e82a-63e8-4aca-bfb5-80ac6a235c5d-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:28:24.826062 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:24.826030 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87e0e82a-63e8-4aca-bfb5-80ac6a235c5d-maas-api-xks-ca" pod "87e0e82a-63e8-4aca-bfb5-80ac6a235c5d" (UID: "87e0e82a-63e8-4aca-bfb5-80ac6a235c5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:28:24.826138 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:24.826124 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e0e82a-63e8-4aca-bfb5-80ac6a235c5d-kube-api-access-xmwpr" pod "87e0e82a-63e8-4aca-bfb5-80ac6a235c5d" (UID: "87e0e82a-63e8-4aca-bfb5-80ac6a235c5d"). InnerVolumeSpecName "kube-api-access-xmwpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:28:24.925239 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:24.925204 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87e0e82a-63e8-4aca-bfb5-80ac6a235c5d-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:28:24.925239 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:24.925236 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmwpr\" (UniqueName: \"kubernetes.io/projected/87e0e82a-63e8-4aca-bfb5-80ac6a235c5d-kube-api-access-xmwpr\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:28:25.259293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:25.253313 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" Oct 30 13:28:25.259507 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:25.259414 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 13:28:25.259507 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:25.259466 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" podUID="6ba44c24-d5d3-461a-b2b3-bca6e225185c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 13:28:25.471348 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:25.471316 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbbe51fe-9b99z" Oct 30 13:28:25.471731 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:25.471315 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbbe51fe-9b99z" event={"ID":"87e0e82a-63e8-4aca-bfb5-80ac6a235c5d","Type":"ContainerDied","Data":"cc758dc374769b0fde78f45707cfcd169b3c58801e08078e31cde4ee437c8a34"} Oct 30 13:28:25.471731 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:25.471444 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc758dc374769b0fde78f45707cfcd169b3c58801e08078e31cde4ee437c8a34" Oct 30 13:28:25.671523 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:25.671495 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1155b409-rwgjs" Oct 30 13:28:25.763741 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:25.763685 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20d5c662-db5a-49ac-acb7-4f8c094297fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d5c662-db5a-49ac-acb7-4f8c094297fd-maas-api-openshift-ca\") pod \"20d5c662-db5a-49ac-acb7-4f8c094297fd\" (UID: \"20d5c662-db5a-49ac-acb7-4f8c094297fd\") " Oct 30 13:28:25.763921 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:25.763752 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20d5c662-db5a-49ac-acb7-4f8c094297fd-kube-api-access-zfc8h\" (UniqueName: \"kubernetes.io/projected/20d5c662-db5a-49ac-acb7-4f8c094297fd-kube-api-access-zfc8h\") pod \"20d5c662-db5a-49ac-acb7-4f8c094297fd\" (UID: \"20d5c662-db5a-49ac-acb7-4f8c094297fd\") " Oct 30 13:28:25.763921 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:25.763828 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20d5c662-db5a-49ac-acb7-4f8c094297fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d5c662-db5a-49ac-acb7-4f8c094297fd-maas-api-xks-ca\") pod \"20d5c662-db5a-49ac-acb7-4f8c094297fd\" (UID: \"20d5c662-db5a-49ac-acb7-4f8c094297fd\") " Oct 30 13:28:25.764530 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:25.764503 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20d5c662-db5a-49ac-acb7-4f8c094297fd-maas-api-openshift-ca" pod "20d5c662-db5a-49ac-acb7-4f8c094297fd" (UID: "20d5c662-db5a-49ac-acb7-4f8c094297fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:28:25.766414 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:25.766369 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20d5c662-db5a-49ac-acb7-4f8c094297fd-maas-api-xks-ca" pod "20d5c662-db5a-49ac-acb7-4f8c094297fd" (UID: "20d5c662-db5a-49ac-acb7-4f8c094297fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:28:25.766632 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:25.766610 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d5c662-db5a-49ac-acb7-4f8c094297fd-kube-api-access-zfc8h" pod "20d5c662-db5a-49ac-acb7-4f8c094297fd" (UID: "20d5c662-db5a-49ac-acb7-4f8c094297fd"). InnerVolumeSpecName "kube-api-access-zfc8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:28:25.866562 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:25.866478 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d5c662-db5a-49ac-acb7-4f8c094297fd-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:28:25.866562 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:25.866511 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d5c662-db5a-49ac-acb7-4f8c094297fd-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:28:25.866562 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:25.866523 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfc8h\" (UniqueName: \"kubernetes.io/projected/20d5c662-db5a-49ac-acb7-4f8c094297fd-kube-api-access-zfc8h\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:28:26.243291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:26.242595 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 13:28:26.243291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:26.242654 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" podUID="6ba44c24-d5d3-461a-b2b3-bca6e225185c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 13:28:26.480979 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:26.480937 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1155b409-rwgjs" Oct 30 13:28:26.480979 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:26.480966 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1155b409-rwgjs" event={"ID":"20d5c662-db5a-49ac-acb7-4f8c094297fd","Type":"ContainerDied","Data":"400c0feed6499659811b8f2aeadf6779ee34045e728e52b59e1afb19e1c2f7d3"} Oct 30 13:28:26.481451 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:26.481015 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="400c0feed6499659811b8f2aeadf6779ee34045e728e52b59e1afb19e1c2f7d3" Oct 30 13:28:27.248292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:27.243766 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 13:28:27.248292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:27.243839 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" podUID="6ba44c24-d5d3-461a-b2b3-bca6e225185c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 13:28:28.247291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:28.239643 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 13:28:28.247291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:28.239707 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" podUID="6ba44c24-d5d3-461a-b2b3-bca6e225185c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 13:28:29.253634 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:29.253593 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 13:28:29.254240 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:29.254206 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" podUID="6ba44c24-d5d3-461a-b2b3-bca6e225185c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 13:28:30.341386 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:30.340149 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" Oct 30 13:28:30.342451 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:30.342383 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" Oct 30 13:28:30.343762 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:30.343734 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" Oct 30 13:28:30.771369 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:30.771338 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5e47c-8xp6x"] Oct 30 13:28:30.774966 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:30.774925 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5e47c-8xp6x"] Oct 30 13:28:31.514388 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:31.514353 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa455cb1-8f65-4233-85fb-d7e651d16140" path="/var/lib/kubelet/pods/aa455cb1-8f65-4233-85fb-d7e651d16140/volumes" Oct 30 13:28:33.260736 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:33.260698 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86bc23cf-ct5rg"] Oct 30 13:28:33.261242 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:33.261228 4931 cpu_manager.go:414] "containerMap: removing container" podUID="87e0e82a-63e8-4aca-bfb5-80ac6a235c5d" containerName="revoke-keys" Oct 30 13:28:33.261307 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:33.261242 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87e0e82a-63e8-4aca-bfb5-80ac6a235c5d" containerName="revoke-keys" Oct 30 13:28:33.261307 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:33.261296 4931 cpu_manager.go:414] "containerMap: removing container" podUID="20d5c662-db5a-49ac-acb7-4f8c094297fd" containerName="revoke-keys" Oct 30 13:28:33.261307 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:33.261302 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20d5c662-db5a-49ac-acb7-4f8c094297fd" containerName="revoke-keys" Oct 30 13:28:33.261412 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:33.261397 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="20d5c662-db5a-49ac-acb7-4f8c094297fd" containerName="revoke-keys" Oct 30 13:28:33.261412 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:33.261410 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="87e0e82a-63e8-4aca-bfb5-80ac6a235c5d" containerName="revoke-keys" Oct 30 13:28:33.265010 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:33.264992 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bc23cf-ct5rg" Oct 30 13:28:33.281744 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:33.281722 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:28:33.295435 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:33.295405 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:28:33.298415 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:33.298382 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d50635b7-b44f-48ff-aea5-dad300e22705-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86bc23cf-ct5rg\" (UID: \"d50635b7-b44f-48ff-aea5-dad300e22705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bc23cf-ct5rg" Oct 30 13:28:33.298607 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:33.298588 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d50635b7-b44f-48ff-aea5-dad300e22705-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86bc23cf-ct5rg\" (UID: \"d50635b7-b44f-48ff-aea5-dad300e22705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bc23cf-ct5rg" Oct 30 13:28:33.298664 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:33.298641 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz7n4\" (UniqueName: \"kubernetes.io/projected/d50635b7-b44f-48ff-aea5-dad300e22705-kube-api-access-dz7n4\") pod \"maas-api-revoke-sub-86bc23cf-ct5rg\" (UID: \"d50635b7-b44f-48ff-aea5-dad300e22705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bc23cf-ct5rg" Oct 30 13:28:33.301733 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:33.301704 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86bc23cf-ct5rg"] Oct 30 13:28:33.399506 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:33.399474 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d50635b7-b44f-48ff-aea5-dad300e22705-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86bc23cf-ct5rg\" (UID: \"d50635b7-b44f-48ff-aea5-dad300e22705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bc23cf-ct5rg" Oct 30 13:28:33.399692 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:33.399525 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz7n4\" (UniqueName: \"kubernetes.io/projected/d50635b7-b44f-48ff-aea5-dad300e22705-kube-api-access-dz7n4\") pod \"maas-api-revoke-sub-86bc23cf-ct5rg\" (UID: \"d50635b7-b44f-48ff-aea5-dad300e22705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bc23cf-ct5rg" Oct 30 13:28:33.399692 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:33.399640 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d50635b7-b44f-48ff-aea5-dad300e22705-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86bc23cf-ct5rg\" (UID: \"d50635b7-b44f-48ff-aea5-dad300e22705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bc23cf-ct5rg" Oct 30 13:28:33.400259 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:33.400238 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d50635b7-b44f-48ff-aea5-dad300e22705-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86bc23cf-ct5rg\" (UID: \"d50635b7-b44f-48ff-aea5-dad300e22705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bc23cf-ct5rg" Oct 30 13:28:33.401772 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:33.401751 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d50635b7-b44f-48ff-aea5-dad300e22705-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86bc23cf-ct5rg\" (UID: \"d50635b7-b44f-48ff-aea5-dad300e22705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bc23cf-ct5rg" Oct 30 13:28:33.424317 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:33.424292 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz7n4\" (UniqueName: \"kubernetes.io/projected/d50635b7-b44f-48ff-aea5-dad300e22705-kube-api-access-dz7n4\") pod \"maas-api-revoke-sub-86bc23cf-ct5rg\" (UID: \"d50635b7-b44f-48ff-aea5-dad300e22705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bc23cf-ct5rg" Oct 30 13:28:33.579230 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:33.579157 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bc23cf-ct5rg" Oct 30 13:28:33.888382 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:28:33.888346 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50635b7_b44f_48ff_aea5_dad300e22705.slice/crio-6aab156b0b4093b11eaeeb8a0a9671c2e2e102fb1ce64486a74d2839fc3eb2c3 WatchSource:0}: Error finding container 6aab156b0b4093b11eaeeb8a0a9671c2e2e102fb1ce64486a74d2839fc3eb2c3: Status 404 returned error can't find the container with id 6aab156b0b4093b11eaeeb8a0a9671c2e2e102fb1ce64486a74d2839fc3eb2c3 Oct 30 13:28:33.888618 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:33.888590 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86bc23cf-ct5rg"] Oct 30 13:28:34.600343 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:34.600300 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bc23cf-ct5rg" event={"ID":"d50635b7-b44f-48ff-aea5-dad300e22705","Type":"ContainerStarted","Data":"d5a34cadd6bc97fc8278b5a8f2c7f6c454577ecaf794b1b45d565973a6111854"} Oct 30 13:28:34.600739 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:34.600387 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bc23cf-ct5rg" event={"ID":"d50635b7-b44f-48ff-aea5-dad300e22705","Type":"ContainerStarted","Data":"6aab156b0b4093b11eaeeb8a0a9671c2e2e102fb1ce64486a74d2839fc3eb2c3"} Oct 30 13:28:34.648907 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:34.648844 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bc23cf-ct5rg" podStartSLOduration=1.6488292530000002 podStartE2EDuration="1.648829253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:34.646131662 +0000 UTC m=+2613.249630108" watchObservedRunningTime="2026-10-30 13:28:34.648829253 +0000 UTC m=+2613.252327724" Oct 30 13:28:35.610180 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:35.610144 4931 generic.go:356] "Generic (PLEG): container finished" podID="d50635b7-b44f-48ff-aea5-dad300e22705" containerID="d5a34cadd6bc97fc8278b5a8f2c7f6c454577ecaf794b1b45d565973a6111854" exitCode=0 Oct 30 13:28:35.610711 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:35.610242 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bc23cf-ct5rg" event={"ID":"d50635b7-b44f-48ff-aea5-dad300e22705","Type":"ContainerDied","Data":"d5a34cadd6bc97fc8278b5a8f2c7f6c454577ecaf794b1b45d565973a6111854"} Oct 30 13:28:36.935152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:36.935125 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bc23cf-ct5rg" Oct 30 13:28:37.062208 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:37.062173 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d50635b7-b44f-48ff-aea5-dad300e22705-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d50635b7-b44f-48ff-aea5-dad300e22705-maas-api-xks-ca\") pod \"d50635b7-b44f-48ff-aea5-dad300e22705\" (UID: \"d50635b7-b44f-48ff-aea5-dad300e22705\") " Oct 30 13:28:37.062383 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:37.062235 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d50635b7-b44f-48ff-aea5-dad300e22705-kube-api-access-dz7n4\" (UniqueName: \"kubernetes.io/projected/d50635b7-b44f-48ff-aea5-dad300e22705-kube-api-access-dz7n4\") pod \"d50635b7-b44f-48ff-aea5-dad300e22705\" (UID: \"d50635b7-b44f-48ff-aea5-dad300e22705\") " Oct 30 13:28:37.062455 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:37.062442 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d50635b7-b44f-48ff-aea5-dad300e22705-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d50635b7-b44f-48ff-aea5-dad300e22705-maas-api-openshift-ca\") pod \"d50635b7-b44f-48ff-aea5-dad300e22705\" (UID: \"d50635b7-b44f-48ff-aea5-dad300e22705\") " Oct 30 13:28:37.062872 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:37.062831 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d50635b7-b44f-48ff-aea5-dad300e22705-maas-api-openshift-ca" pod "d50635b7-b44f-48ff-aea5-dad300e22705" (UID: "d50635b7-b44f-48ff-aea5-dad300e22705"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:28:37.063008 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:37.062967 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d50635b7-b44f-48ff-aea5-dad300e22705-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:28:37.064502 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:37.064475 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d50635b7-b44f-48ff-aea5-dad300e22705-maas-api-xks-ca" pod "d50635b7-b44f-48ff-aea5-dad300e22705" (UID: "d50635b7-b44f-48ff-aea5-dad300e22705"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:28:37.064602 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:37.064497 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d50635b7-b44f-48ff-aea5-dad300e22705-kube-api-access-dz7n4" pod "d50635b7-b44f-48ff-aea5-dad300e22705" (UID: "d50635b7-b44f-48ff-aea5-dad300e22705"). InnerVolumeSpecName "kube-api-access-dz7n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:28:37.165343 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:37.164530 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d50635b7-b44f-48ff-aea5-dad300e22705-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:28:37.165343 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:37.164566 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz7n4\" (UniqueName: \"kubernetes.io/projected/d50635b7-b44f-48ff-aea5-dad300e22705-kube-api-access-dz7n4\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:28:37.352624 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:28:37.352590 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:28:37.626343 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:37.626315 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bc23cf-ct5rg" Oct 30 13:28:37.626512 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:37.626309 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bc23cf-ct5rg" event={"ID":"d50635b7-b44f-48ff-aea5-dad300e22705","Type":"ContainerDied","Data":"6aab156b0b4093b11eaeeb8a0a9671c2e2e102fb1ce64486a74d2839fc3eb2c3"} Oct 30 13:28:37.626512 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:37.626450 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6aab156b0b4093b11eaeeb8a0a9671c2e2e102fb1ce64486a74d2839fc3eb2c3" Oct 30 13:28:43.892750 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:43.892711 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-475f32cc-smnnr"] Oct 30 13:28:43.893719 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:43.893280 4931 cpu_manager.go:414] "containerMap: removing container" podUID="d50635b7-b44f-48ff-aea5-dad300e22705" containerName="revoke-keys" Oct 30 13:28:43.893719 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:43.893293 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d50635b7-b44f-48ff-aea5-dad300e22705" containerName="revoke-keys" Oct 30 13:28:43.893719 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:43.893396 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="d50635b7-b44f-48ff-aea5-dad300e22705" containerName="revoke-keys" Oct 30 13:28:43.897058 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:43.897041 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475f32cc-smnnr" Oct 30 13:28:43.924122 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:43.924097 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-475f32cc-smnnr"] Oct 30 13:28:43.932561 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:43.932530 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:28:43.932683 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:43.932603 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:28:44.061820 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:44.061783 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09283563-d7cc-43f6-b34a-e683eae58317-maas-api-xks-ca\") pod \"maas-api-revoke-sub-475f32cc-smnnr\" (UID: \"09283563-d7cc-43f6-b34a-e683eae58317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475f32cc-smnnr" Oct 30 13:28:44.061999 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:44.061908 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09283563-d7cc-43f6-b34a-e683eae58317-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-475f32cc-smnnr\" (UID: \"09283563-d7cc-43f6-b34a-e683eae58317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475f32cc-smnnr" Oct 30 13:28:44.061999 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:44.061974 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr82h\" (UniqueName: \"kubernetes.io/projected/09283563-d7cc-43f6-b34a-e683eae58317-kube-api-access-xr82h\") pod \"maas-api-revoke-sub-475f32cc-smnnr\" (UID: \"09283563-d7cc-43f6-b34a-e683eae58317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475f32cc-smnnr" Oct 30 13:28:44.163024 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:44.162938 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09283563-d7cc-43f6-b34a-e683eae58317-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-475f32cc-smnnr\" (UID: \"09283563-d7cc-43f6-b34a-e683eae58317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475f32cc-smnnr" Oct 30 13:28:44.163024 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:44.162989 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr82h\" (UniqueName: \"kubernetes.io/projected/09283563-d7cc-43f6-b34a-e683eae58317-kube-api-access-xr82h\") pod \"maas-api-revoke-sub-475f32cc-smnnr\" (UID: \"09283563-d7cc-43f6-b34a-e683eae58317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475f32cc-smnnr" Oct 30 13:28:44.163237 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:44.163204 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09283563-d7cc-43f6-b34a-e683eae58317-maas-api-xks-ca\") pod \"maas-api-revoke-sub-475f32cc-smnnr\" (UID: \"09283563-d7cc-43f6-b34a-e683eae58317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475f32cc-smnnr" Oct 30 13:28:44.163832 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:44.163801 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09283563-d7cc-43f6-b34a-e683eae58317-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-475f32cc-smnnr\" (UID: \"09283563-d7cc-43f6-b34a-e683eae58317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475f32cc-smnnr" Oct 30 13:28:44.166073 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:44.166051 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09283563-d7cc-43f6-b34a-e683eae58317-maas-api-xks-ca\") pod \"maas-api-revoke-sub-475f32cc-smnnr\" (UID: \"09283563-d7cc-43f6-b34a-e683eae58317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475f32cc-smnnr" Oct 30 13:28:44.188477 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:44.188457 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr82h\" (UniqueName: \"kubernetes.io/projected/09283563-d7cc-43f6-b34a-e683eae58317-kube-api-access-xr82h\") pod \"maas-api-revoke-sub-475f32cc-smnnr\" (UID: \"09283563-d7cc-43f6-b34a-e683eae58317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475f32cc-smnnr" Oct 30 13:28:44.211194 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:44.211173 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475f32cc-smnnr" Oct 30 13:28:44.524807 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:28:44.524776 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09283563_d7cc_43f6_b34a_e683eae58317.slice/crio-91be7cfa03c3cbc9f027f8147a03fd95a52de91c382ab595e4ce4f73ece4755e WatchSource:0}: Error finding container 91be7cfa03c3cbc9f027f8147a03fd95a52de91c382ab595e4ce4f73ece4755e: Status 404 returned error can't find the container with id 91be7cfa03c3cbc9f027f8147a03fd95a52de91c382ab595e4ce4f73ece4755e Oct 30 13:28:44.525044 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:44.525025 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-475f32cc-smnnr"] Oct 30 13:28:44.683542 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:44.683501 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475f32cc-smnnr" event={"ID":"09283563-d7cc-43f6-b34a-e683eae58317","Type":"ContainerStarted","Data":"91be7cfa03c3cbc9f027f8147a03fd95a52de91c382ab595e4ce4f73ece4755e"} Oct 30 13:28:45.708421 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:45.708382 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475f32cc-smnnr" event={"ID":"09283563-d7cc-43f6-b34a-e683eae58317","Type":"ContainerStarted","Data":"7452905f511b68dc31af6a9e8efa75b8e87cbabbd5380daeb75144c6b9f1dfa1"} Oct 30 13:28:46.717233 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:46.717201 4931 generic.go:356] "Generic (PLEG): container finished" podID="09283563-d7cc-43f6-b34a-e683eae58317" containerID="7452905f511b68dc31af6a9e8efa75b8e87cbabbd5380daeb75144c6b9f1dfa1" exitCode=0 Oct 30 13:28:46.717661 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:46.717317 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475f32cc-smnnr" event={"ID":"09283563-d7cc-43f6-b34a-e683eae58317","Type":"ContainerDied","Data":"7452905f511b68dc31af6a9e8efa75b8e87cbabbd5380daeb75144c6b9f1dfa1"} Oct 30 13:28:46.901335 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:46.901306 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475f32cc-smnnr" Oct 30 13:28:47.034639 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:47.034550 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09283563-d7cc-43f6-b34a-e683eae58317-kube-api-access-xr82h\" (UniqueName: \"kubernetes.io/projected/09283563-d7cc-43f6-b34a-e683eae58317-kube-api-access-xr82h\") pod \"09283563-d7cc-43f6-b34a-e683eae58317\" (UID: \"09283563-d7cc-43f6-b34a-e683eae58317\") " Oct 30 13:28:47.034817 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:47.034780 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09283563-d7cc-43f6-b34a-e683eae58317-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09283563-d7cc-43f6-b34a-e683eae58317-maas-api-openshift-ca\") pod \"09283563-d7cc-43f6-b34a-e683eae58317\" (UID: \"09283563-d7cc-43f6-b34a-e683eae58317\") " Oct 30 13:28:47.034905 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:47.034824 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09283563-d7cc-43f6-b34a-e683eae58317-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09283563-d7cc-43f6-b34a-e683eae58317-maas-api-xks-ca\") pod \"09283563-d7cc-43f6-b34a-e683eae58317\" (UID: \"09283563-d7cc-43f6-b34a-e683eae58317\") " Oct 30 13:28:47.035258 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:47.035214 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09283563-d7cc-43f6-b34a-e683eae58317-maas-api-openshift-ca" pod "09283563-d7cc-43f6-b34a-e683eae58317" (UID: "09283563-d7cc-43f6-b34a-e683eae58317"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:28:47.035425 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:47.035391 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09283563-d7cc-43f6-b34a-e683eae58317-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:28:47.037005 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:47.036966 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09283563-d7cc-43f6-b34a-e683eae58317-kube-api-access-xr82h" pod "09283563-d7cc-43f6-b34a-e683eae58317" (UID: "09283563-d7cc-43f6-b34a-e683eae58317"). InnerVolumeSpecName "kube-api-access-xr82h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:28:47.037473 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:47.037450 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09283563-d7cc-43f6-b34a-e683eae58317-maas-api-xks-ca" pod "09283563-d7cc-43f6-b34a-e683eae58317" (UID: "09283563-d7cc-43f6-b34a-e683eae58317"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:28:47.136517 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:47.136483 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09283563-d7cc-43f6-b34a-e683eae58317-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:28:47.136517 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:47.136517 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr82h\" (UniqueName: \"kubernetes.io/projected/09283563-d7cc-43f6-b34a-e683eae58317-kube-api-access-xr82h\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:28:47.725375 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:47.725342 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475f32cc-smnnr" Oct 30 13:28:47.725375 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:47.725372 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475f32cc-smnnr" event={"ID":"09283563-d7cc-43f6-b34a-e683eae58317","Type":"ContainerDied","Data":"91be7cfa03c3cbc9f027f8147a03fd95a52de91c382ab595e4ce4f73ece4755e"} Oct 30 13:28:47.725838 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:47.725428 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91be7cfa03c3cbc9f027f8147a03fd95a52de91c382ab595e4ce4f73ece4755e" Oct 30 13:28:54.263689 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.263650 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg"] Oct 30 13:28:54.264230 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.264216 4931 cpu_manager.go:414] "containerMap: removing container" podUID="09283563-d7cc-43f6-b34a-e683eae58317" containerName="revoke-keys" Oct 30 13:28:54.264339 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.264232 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09283563-d7cc-43f6-b34a-e683eae58317" containerName="revoke-keys" Oct 30 13:28:54.264384 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.264362 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="09283563-d7cc-43f6-b34a-e683eae58317" containerName="revoke-keys" Oct 30 13:28:54.268993 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.268969 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" Oct 30 13:28:54.305798 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.305767 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg"] Oct 30 13:28:54.333903 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.333864 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a110d96-ba95-4582-9e51-321f91ea6510-istio-podinfo\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" Oct 30 13:28:54.334077 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.333941 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a110d96-ba95-4582-9e51-321f91ea6510-istiod-ca-cert\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" Oct 30 13:28:54.334077 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.334024 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a110d96-ba95-4582-9e51-321f91ea6510-istio-data\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" Oct 30 13:28:54.334077 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.334049 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a110d96-ba95-4582-9e51-321f91ea6510-workload-certs\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" Oct 30 13:28:54.334221 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.334131 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a110d96-ba95-4582-9e51-321f91ea6510-credential-socket\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" Oct 30 13:28:54.334221 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.334168 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a110d96-ba95-4582-9e51-321f91ea6510-workload-socket\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" Oct 30 13:28:54.334404 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.334261 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a110d96-ba95-4582-9e51-321f91ea6510-istio-token\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" Oct 30 13:28:54.334454 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.334398 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a110d96-ba95-4582-9e51-321f91ea6510-istio-envoy\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" Oct 30 13:28:54.334516 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.334506 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnzdl\" (UniqueName: \"kubernetes.io/projected/8a110d96-ba95-4582-9e51-321f91ea6510-kube-api-access-vnzdl\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" Oct 30 13:28:54.362239 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.362212 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-8a5d5e-openshift-default-dockercfg-t586t\"" Oct 30 13:28:54.435468 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.435430 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a110d96-ba95-4582-9e51-321f91ea6510-istio-podinfo\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" Oct 30 13:28:54.435627 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.435474 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a110d96-ba95-4582-9e51-321f91ea6510-istiod-ca-cert\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" Oct 30 13:28:54.435627 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.435517 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a110d96-ba95-4582-9e51-321f91ea6510-istio-data\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" Oct 30 13:28:54.435627 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.435547 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a110d96-ba95-4582-9e51-321f91ea6510-workload-certs\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" Oct 30 13:28:54.435627 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.435619 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a110d96-ba95-4582-9e51-321f91ea6510-credential-socket\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" Oct 30 13:28:54.435852 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.435654 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a110d96-ba95-4582-9e51-321f91ea6510-workload-socket\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" Oct 30 13:28:54.435852 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.435684 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a110d96-ba95-4582-9e51-321f91ea6510-istio-token\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" Oct 30 13:28:54.435852 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.435764 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a110d96-ba95-4582-9e51-321f91ea6510-istio-envoy\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" Oct 30 13:28:54.436000 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.435871 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnzdl\" (UniqueName: \"kubernetes.io/projected/8a110d96-ba95-4582-9e51-321f91ea6510-kube-api-access-vnzdl\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" Oct 30 13:28:54.436369 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.436335 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a110d96-ba95-4582-9e51-321f91ea6510-istiod-ca-cert\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" Oct 30 13:28:54.436511 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.436410 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a110d96-ba95-4582-9e51-321f91ea6510-istio-data\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" Oct 30 13:28:54.436511 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.436463 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a110d96-ba95-4582-9e51-321f91ea6510-credential-socket\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" Oct 30 13:28:54.436633 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.436543 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a110d96-ba95-4582-9e51-321f91ea6510-workload-certs\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" Oct 30 13:28:54.436685 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.436669 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a110d96-ba95-4582-9e51-321f91ea6510-workload-socket\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" Oct 30 13:28:54.438645 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.438622 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a110d96-ba95-4582-9e51-321f91ea6510-istio-envoy\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" Oct 30 13:28:54.438893 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.438865 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a110d96-ba95-4582-9e51-321f91ea6510-istio-podinfo\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" Oct 30 13:28:54.460104 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.460073 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a110d96-ba95-4582-9e51-321f91ea6510-istio-token\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" Oct 30 13:28:54.460444 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.460418 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnzdl\" (UniqueName: \"kubernetes.io/projected/8a110d96-ba95-4582-9e51-321f91ea6510-kube-api-access-vnzdl\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" Oct 30 13:28:54.585743 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.585671 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" Oct 30 13:28:54.913591 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.913555 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg"] Oct 30 13:28:54.913979 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:28:54.913952 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a110d96_ba95_4582_9e51_321f91ea6510.slice/crio-17f493d53e5b456fb48d64bfb72ac8808078a26cd3a2fd88a3223fd2f832415d WatchSource:0}: Error finding container 17f493d53e5b456fb48d64bfb72ac8808078a26cd3a2fd88a3223fd2f832415d: Status 404 returned error can't find the container with id 17f493d53e5b456fb48d64bfb72ac8808078a26cd3a2fd88a3223fd2f832415d Oct 30 13:28:54.916662 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.916629 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:28:54.916745 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.916693 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:28:54.916745 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:54.916731 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:28:55.812123 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:55.812087 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" event={"ID":"8a110d96-ba95-4582-9e51-321f91ea6510","Type":"ContainerStarted","Data":"59bc6d872406f1fd6f13b11ce1850fd8f7e77be1336e8994d4b5ceb9a3dccdd7"} Oct 30 13:28:55.812584 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:55.812143 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" event={"ID":"8a110d96-ba95-4582-9e51-321f91ea6510","Type":"ContainerStarted","Data":"17f493d53e5b456fb48d64bfb72ac8808078a26cd3a2fd88a3223fd2f832415d"} Oct 30 13:28:55.866849 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:55.866795 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" podStartSLOduration=1.86678097 podStartE2EDuration="1.86678097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:55.859211836 +0000 UTC m=+2634.462710311" watchObservedRunningTime="2026-10-30 13:28:55.86678097 +0000 UTC m=+2634.470279476" Oct 30 13:28:56.586388 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:56.586349 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" Oct 30 13:28:56.588310 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:56.588253 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 13:28:56.588442 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:56.588330 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" podUID="8a110d96-ba95-4582-9e51-321f91ea6510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 13:28:57.586870 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:57.586830 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 13:28:57.587380 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:57.586892 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" podUID="8a110d96-ba95-4582-9e51-321f91ea6510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 13:28:58.586719 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:58.586682 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 13:28:58.586925 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:58.586744 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" podUID="8a110d96-ba95-4582-9e51-321f91ea6510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 13:28:59.586189 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:59.586151 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 13:28:59.586406 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:28:59.586207 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" podUID="8a110d96-ba95-4582-9e51-321f91ea6510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 13:29:00.586914 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:00.586879 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 13:29:00.587334 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:00.586944 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" podUID="8a110d96-ba95-4582-9e51-321f91ea6510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 13:29:01.590676 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:01.590643 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" Oct 30 13:29:01.591185 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:01.591084 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" Oct 30 13:29:01.591906 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:01.591882 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" Oct 30 13:29:07.353393 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:29:07.353350 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:29:15.397456 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:15.397375 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:29:18.408848 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:18.408802 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs"] Oct 30 13:29:27.665331 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:27.665294 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1efa867-tms6c"] Oct 30 13:29:27.671023 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:27.671003 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1efa867-tms6c" Oct 30 13:29:27.691998 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:27.691969 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1efa867-tms6c"] Oct 30 13:29:27.699471 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:27.699421 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:29:27.699633 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:27.699497 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:29:27.803219 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:27.803175 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4mgw\" (UniqueName: \"kubernetes.io/projected/2be096ed-3a80-4449-8ea1-3281d1fa5ba5-kube-api-access-r4mgw\") pod \"maas-api-revoke-sub-a1efa867-tms6c\" (UID: \"2be096ed-3a80-4449-8ea1-3281d1fa5ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1efa867-tms6c" Oct 30 13:29:27.803433 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:27.803371 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2be096ed-3a80-4449-8ea1-3281d1fa5ba5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1efa867-tms6c\" (UID: \"2be096ed-3a80-4449-8ea1-3281d1fa5ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1efa867-tms6c" Oct 30 13:29:27.803481 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:27.803467 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2be096ed-3a80-4449-8ea1-3281d1fa5ba5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1efa867-tms6c\" (UID: \"2be096ed-3a80-4449-8ea1-3281d1fa5ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1efa867-tms6c" Oct 30 13:29:27.904560 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:27.904516 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2be096ed-3a80-4449-8ea1-3281d1fa5ba5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1efa867-tms6c\" (UID: \"2be096ed-3a80-4449-8ea1-3281d1fa5ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1efa867-tms6c" Oct 30 13:29:27.904745 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:27.904640 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2be096ed-3a80-4449-8ea1-3281d1fa5ba5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1efa867-tms6c\" (UID: \"2be096ed-3a80-4449-8ea1-3281d1fa5ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1efa867-tms6c" Oct 30 13:29:27.904788 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:27.904761 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4mgw\" (UniqueName: \"kubernetes.io/projected/2be096ed-3a80-4449-8ea1-3281d1fa5ba5-kube-api-access-r4mgw\") pod \"maas-api-revoke-sub-a1efa867-tms6c\" (UID: \"2be096ed-3a80-4449-8ea1-3281d1fa5ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1efa867-tms6c" Oct 30 13:29:27.905346 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:27.905318 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2be096ed-3a80-4449-8ea1-3281d1fa5ba5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1efa867-tms6c\" (UID: \"2be096ed-3a80-4449-8ea1-3281d1fa5ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1efa867-tms6c" Oct 30 13:29:27.907120 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:27.907091 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2be096ed-3a80-4449-8ea1-3281d1fa5ba5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1efa867-tms6c\" (UID: \"2be096ed-3a80-4449-8ea1-3281d1fa5ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1efa867-tms6c" Oct 30 13:29:27.929542 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:27.929483 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4mgw\" (UniqueName: \"kubernetes.io/projected/2be096ed-3a80-4449-8ea1-3281d1fa5ba5-kube-api-access-r4mgw\") pod \"maas-api-revoke-sub-a1efa867-tms6c\" (UID: \"2be096ed-3a80-4449-8ea1-3281d1fa5ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1efa867-tms6c" Oct 30 13:29:27.985201 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:27.985152 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1efa867-tms6c" Oct 30 13:29:28.296760 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:29:28.296723 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2be096ed_3a80_4449_8ea1_3281d1fa5ba5.slice/crio-5b5328bda3fb9b2ca60dff0d987893e3ac632653ba38335df7a16f8d52414266 WatchSource:0}: Error finding container 5b5328bda3fb9b2ca60dff0d987893e3ac632653ba38335df7a16f8d52414266: Status 404 returned error can't find the container with id 5b5328bda3fb9b2ca60dff0d987893e3ac632653ba38335df7a16f8d52414266 Oct 30 13:29:28.296955 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:28.296932 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1efa867-tms6c"] Oct 30 13:29:29.285440 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:29.285412 4931 generic.go:356] "Generic (PLEG): container finished" podID="2be096ed-3a80-4449-8ea1-3281d1fa5ba5" containerID="c9c418151585261061be2736ce244777dab22d4e60b5ebcb232f19e9397384b0" exitCode=0 Oct 30 13:29:29.285811 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:29.285491 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1efa867-tms6c" event={"ID":"2be096ed-3a80-4449-8ea1-3281d1fa5ba5","Type":"ContainerDied","Data":"c9c418151585261061be2736ce244777dab22d4e60b5ebcb232f19e9397384b0"} Oct 30 13:29:29.285811 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:29.285556 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1efa867-tms6c" event={"ID":"2be096ed-3a80-4449-8ea1-3281d1fa5ba5","Type":"ContainerStarted","Data":"5b5328bda3fb9b2ca60dff0d987893e3ac632653ba38335df7a16f8d52414266"} Oct 30 13:29:30.444323 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:30.444299 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1efa867-tms6c" Oct 30 13:29:30.544438 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:30.544350 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2be096ed-3a80-4449-8ea1-3281d1fa5ba5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2be096ed-3a80-4449-8ea1-3281d1fa5ba5-maas-api-xks-ca\") pod \"2be096ed-3a80-4449-8ea1-3281d1fa5ba5\" (UID: \"2be096ed-3a80-4449-8ea1-3281d1fa5ba5\") " Oct 30 13:29:30.544628 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:30.544442 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2be096ed-3a80-4449-8ea1-3281d1fa5ba5-kube-api-access-r4mgw\" (UniqueName: \"kubernetes.io/projected/2be096ed-3a80-4449-8ea1-3281d1fa5ba5-kube-api-access-r4mgw\") pod \"2be096ed-3a80-4449-8ea1-3281d1fa5ba5\" (UID: \"2be096ed-3a80-4449-8ea1-3281d1fa5ba5\") " Oct 30 13:29:30.544628 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:30.544562 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2be096ed-3a80-4449-8ea1-3281d1fa5ba5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2be096ed-3a80-4449-8ea1-3281d1fa5ba5-maas-api-openshift-ca\") pod \"2be096ed-3a80-4449-8ea1-3281d1fa5ba5\" (UID: \"2be096ed-3a80-4449-8ea1-3281d1fa5ba5\") " Oct 30 13:29:30.545213 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:30.545179 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2be096ed-3a80-4449-8ea1-3281d1fa5ba5-maas-api-openshift-ca" pod "2be096ed-3a80-4449-8ea1-3281d1fa5ba5" (UID: "2be096ed-3a80-4449-8ea1-3281d1fa5ba5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:29:30.545736 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:30.545709 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2be096ed-3a80-4449-8ea1-3281d1fa5ba5-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:29:30.546918 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:30.546893 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2be096ed-3a80-4449-8ea1-3281d1fa5ba5-kube-api-access-r4mgw" pod "2be096ed-3a80-4449-8ea1-3281d1fa5ba5" (UID: "2be096ed-3a80-4449-8ea1-3281d1fa5ba5"). InnerVolumeSpecName "kube-api-access-r4mgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:30.547613 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:30.547594 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2be096ed-3a80-4449-8ea1-3281d1fa5ba5-maas-api-xks-ca" pod "2be096ed-3a80-4449-8ea1-3281d1fa5ba5" (UID: "2be096ed-3a80-4449-8ea1-3281d1fa5ba5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:29:30.647581 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:30.647548 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4mgw\" (UniqueName: \"kubernetes.io/projected/2be096ed-3a80-4449-8ea1-3281d1fa5ba5-kube-api-access-r4mgw\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:29:30.647581 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:30.647581 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2be096ed-3a80-4449-8ea1-3281d1fa5ba5-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:29:31.311037 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:31.311006 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1efa867-tms6c" Oct 30 13:29:31.311218 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:31.311038 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1efa867-tms6c" event={"ID":"2be096ed-3a80-4449-8ea1-3281d1fa5ba5","Type":"ContainerDied","Data":"5b5328bda3fb9b2ca60dff0d987893e3ac632653ba38335df7a16f8d52414266"} Oct 30 13:29:31.311218 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:31.311086 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b5328bda3fb9b2ca60dff0d987893e3ac632653ba38335df7a16f8d52414266" Oct 30 13:29:33.430876 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:33.430830 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs" podUID="8ca147ee-5965-483a-a4a2-7383326e09b0" containerName="main" containerID="cri-o://6e34d813fb649724517a3881a302b371dc1d7b84099b008e6eb7b72949a6fe6d" gracePeriod=45 Oct 30 13:29:33.433838 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:33.433808 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs" podUID="8ca147ee-5965-483a-a4a2-7383326e09b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.167:8000/ready\": dial tcp 10.132.0.167:8000: connect: connection reset by peer" Oct 30 13:29:33.710755 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:33.710730 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs" Oct 30 13:29:33.811398 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:33.811368 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ca147ee-5965-483a-a4a2-7383326e09b0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ca147ee-5965-483a-a4a2-7383326e09b0-tmp-dir\") pod \"8ca147ee-5965-483a-a4a2-7383326e09b0\" (UID: \"8ca147ee-5965-483a-a4a2-7383326e09b0\") " Oct 30 13:29:33.811398 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:33.811406 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ca147ee-5965-483a-a4a2-7383326e09b0-tls-certs\" (UniqueName: \"kubernetes.io/secret/8ca147ee-5965-483a-a4a2-7383326e09b0-tls-certs\") pod \"8ca147ee-5965-483a-a4a2-7383326e09b0\" (UID: \"8ca147ee-5965-483a-a4a2-7383326e09b0\") " Oct 30 13:29:33.811621 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:33.811435 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ca147ee-5965-483a-a4a2-7383326e09b0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ca147ee-5965-483a-a4a2-7383326e09b0-model-cache\") pod \"8ca147ee-5965-483a-a4a2-7383326e09b0\" (UID: \"8ca147ee-5965-483a-a4a2-7383326e09b0\") " Oct 30 13:29:33.811621 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:33.811562 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ca147ee-5965-483a-a4a2-7383326e09b0-home\" (UniqueName: \"kubernetes.io/empty-dir/8ca147ee-5965-483a-a4a2-7383326e09b0-home\") pod \"8ca147ee-5965-483a-a4a2-7383326e09b0\" (UID: \"8ca147ee-5965-483a-a4a2-7383326e09b0\") " Oct 30 13:29:33.811751 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:33.811630 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ca147ee-5965-483a-a4a2-7383326e09b0-kube-api-access-t6r4v\" (UniqueName: \"kubernetes.io/projected/8ca147ee-5965-483a-a4a2-7383326e09b0-kube-api-access-t6r4v\") pod \"8ca147ee-5965-483a-a4a2-7383326e09b0\" (UID: \"8ca147ee-5965-483a-a4a2-7383326e09b0\") " Oct 30 13:29:33.811751 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:33.811676 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ca147ee-5965-483a-a4a2-7383326e09b0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ca147ee-5965-483a-a4a2-7383326e09b0-dshm\") pod \"8ca147ee-5965-483a-a4a2-7383326e09b0\" (UID: \"8ca147ee-5965-483a-a4a2-7383326e09b0\") " Oct 30 13:29:33.811751 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:33.811678 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ca147ee-5965-483a-a4a2-7383326e09b0-tmp-dir" pod "8ca147ee-5965-483a-a4a2-7383326e09b0" (UID: "8ca147ee-5965-483a-a4a2-7383326e09b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:29:33.811751 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:33.811691 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ca147ee-5965-483a-a4a2-7383326e09b0-model-cache" pod "8ca147ee-5965-483a-a4a2-7383326e09b0" (UID: "8ca147ee-5965-483a-a4a2-7383326e09b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:29:33.811965 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:33.811782 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ca147ee-5965-483a-a4a2-7383326e09b0-home" pod "8ca147ee-5965-483a-a4a2-7383326e09b0" (UID: "8ca147ee-5965-483a-a4a2-7383326e09b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:29:33.812159 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:33.812133 4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ca147ee-5965-483a-a4a2-7383326e09b0-tmp-dir\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:29:33.812159 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:33.812162 4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ca147ee-5965-483a-a4a2-7383326e09b0-model-cache\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:29:33.812391 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:33.812175 4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ca147ee-5965-483a-a4a2-7383326e09b0-home\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:29:33.813809 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:33.813782 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ca147ee-5965-483a-a4a2-7383326e09b0-dshm" pod "8ca147ee-5965-483a-a4a2-7383326e09b0" (UID: "8ca147ee-5965-483a-a4a2-7383326e09b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:29:33.813809 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:33.813800 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ca147ee-5965-483a-a4a2-7383326e09b0-tls-certs" pod "8ca147ee-5965-483a-a4a2-7383326e09b0" (UID: "8ca147ee-5965-483a-a4a2-7383326e09b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:29:33.813947 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:33.813862 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ca147ee-5965-483a-a4a2-7383326e09b0-kube-api-access-t6r4v" pod "8ca147ee-5965-483a-a4a2-7383326e09b0" (UID: "8ca147ee-5965-483a-a4a2-7383326e09b0"). InnerVolumeSpecName "kube-api-access-t6r4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:33.913371 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:33.913338 4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ca147ee-5965-483a-a4a2-7383326e09b0-tls-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:29:33.913371 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:33.913374 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6r4v\" (UniqueName: \"kubernetes.io/projected/8ca147ee-5965-483a-a4a2-7383326e09b0-kube-api-access-t6r4v\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:29:33.913612 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:33.913389 4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ca147ee-5965-483a-a4a2-7383326e09b0-dshm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:29:34.334045 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:34.334010 4931 generic.go:356] "Generic (PLEG): container finished" podID="8ca147ee-5965-483a-a4a2-7383326e09b0" containerID="6e34d813fb649724517a3881a302b371dc1d7b84099b008e6eb7b72949a6fe6d" exitCode=0 Oct 30 13:29:34.334216 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:34.334063 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs" Oct 30 13:29:34.334216 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:34.334086 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs" event={"ID":"8ca147ee-5965-483a-a4a2-7383326e09b0","Type":"ContainerDied","Data":"6e34d813fb649724517a3881a302b371dc1d7b84099b008e6eb7b72949a6fe6d"} Oct 30 13:29:34.334216 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:34.334128 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs" event={"ID":"8ca147ee-5965-483a-a4a2-7383326e09b0","Type":"ContainerDied","Data":"12a314a28894dbf5244ff95e6510ea173ee247ee61514c7e21d359e577004f7c"} Oct 30 13:29:34.334216 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:34.334148 4931 scope.go:122] "RemoveContainer" containerID="6e34d813fb649724517a3881a302b371dc1d7b84099b008e6eb7b72949a6fe6d" Oct 30 13:29:34.348559 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:34.348535 4931 scope.go:122] "RemoveContainer" containerID="6e34d813fb649724517a3881a302b371dc1d7b84099b008e6eb7b72949a6fe6d" Oct 30 13:29:34.348894 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:29:34.348867 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e34d813fb649724517a3881a302b371dc1d7b84099b008e6eb7b72949a6fe6d\": container with ID starting with 6e34d813fb649724517a3881a302b371dc1d7b84099b008e6eb7b72949a6fe6d not found: ID does not exist" containerID="6e34d813fb649724517a3881a302b371dc1d7b84099b008e6eb7b72949a6fe6d" Oct 30 13:29:34.349000 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:34.348905 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e34d813fb649724517a3881a302b371dc1d7b84099b008e6eb7b72949a6fe6d"} err="failed to get container status \"6e34d813fb649724517a3881a302b371dc1d7b84099b008e6eb7b72949a6fe6d\": rpc error: code = NotFound desc = could not find container \"6e34d813fb649724517a3881a302b371dc1d7b84099b008e6eb7b72949a6fe6d\": container with ID starting with 6e34d813fb649724517a3881a302b371dc1d7b84099b008e6eb7b72949a6fe6d not found: ID does not exist" Oct 30 13:29:34.396599 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:34.396574 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs"] Oct 30 13:29:34.416057 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:34.416025 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-7a71ca/auth-test-model-7a71ca-kserve-5cc747cf9c-hl5hs"] Oct 30 13:29:35.514096 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:35.514062 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ca147ee-5965-483a-a4a2-7383326e09b0" path="/var/lib/kubelet/pods/8ca147ee-5965-483a-a4a2-7383326e09b0/volumes" Oct 30 13:29:37.353113 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:29:37.353077 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:29:38.003804 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:38.003765 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d9f4c53-htbbm"] Oct 30 13:29:38.004481 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:38.004456 4931 cpu_manager.go:414] "containerMap: removing container" podUID="2be096ed-3a80-4449-8ea1-3281d1fa5ba5" containerName="revoke-keys" Oct 30 13:29:38.004481 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:38.004481 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2be096ed-3a80-4449-8ea1-3281d1fa5ba5" containerName="revoke-keys" Oct 30 13:29:38.004685 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:38.004509 4931 cpu_manager.go:414] "containerMap: removing container" podUID="8ca147ee-5965-483a-a4a2-7383326e09b0" containerName="main" Oct 30 13:29:38.004685 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:38.004517 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ca147ee-5965-483a-a4a2-7383326e09b0" containerName="main" Oct 30 13:29:38.004685 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:38.004670 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="2be096ed-3a80-4449-8ea1-3281d1fa5ba5" containerName="revoke-keys" Oct 30 13:29:38.004842 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:38.004686 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ca147ee-5965-483a-a4a2-7383326e09b0" containerName="main" Oct 30 13:29:38.009771 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:38.009749 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9f4c53-htbbm" Oct 30 13:29:38.029572 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:38.029549 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d9f4c53-htbbm"] Oct 30 13:29:38.038654 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:38.038630 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:29:38.038783 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:38.038686 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:29:38.180501 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:38.180467 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzfp8\" (UniqueName: \"kubernetes.io/projected/3324d5df-e2ef-46ab-89b2-3b64126732b9-kube-api-access-lzfp8\") pod \"maas-api-revoke-sub-6d9f4c53-htbbm\" (UID: \"3324d5df-e2ef-46ab-89b2-3b64126732b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9f4c53-htbbm" Oct 30 13:29:38.180705 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:38.180591 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3324d5df-e2ef-46ab-89b2-3b64126732b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d9f4c53-htbbm\" (UID: \"3324d5df-e2ef-46ab-89b2-3b64126732b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9f4c53-htbbm" Oct 30 13:29:38.180705 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:38.180643 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3324d5df-e2ef-46ab-89b2-3b64126732b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d9f4c53-htbbm\" (UID: \"3324d5df-e2ef-46ab-89b2-3b64126732b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9f4c53-htbbm" Oct 30 13:29:38.281966 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:38.281898 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzfp8\" (UniqueName: \"kubernetes.io/projected/3324d5df-e2ef-46ab-89b2-3b64126732b9-kube-api-access-lzfp8\") pod \"maas-api-revoke-sub-6d9f4c53-htbbm\" (UID: \"3324d5df-e2ef-46ab-89b2-3b64126732b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9f4c53-htbbm" Oct 30 13:29:38.282103 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:38.281989 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3324d5df-e2ef-46ab-89b2-3b64126732b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d9f4c53-htbbm\" (UID: \"3324d5df-e2ef-46ab-89b2-3b64126732b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9f4c53-htbbm" Oct 30 13:29:38.282103 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:38.282020 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3324d5df-e2ef-46ab-89b2-3b64126732b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d9f4c53-htbbm\" (UID: \"3324d5df-e2ef-46ab-89b2-3b64126732b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9f4c53-htbbm" Oct 30 13:29:38.282683 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:38.282657 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3324d5df-e2ef-46ab-89b2-3b64126732b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d9f4c53-htbbm\" (UID: \"3324d5df-e2ef-46ab-89b2-3b64126732b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9f4c53-htbbm" Oct 30 13:29:38.284382 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:38.284363 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3324d5df-e2ef-46ab-89b2-3b64126732b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d9f4c53-htbbm\" (UID: \"3324d5df-e2ef-46ab-89b2-3b64126732b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9f4c53-htbbm" Oct 30 13:29:38.306680 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:38.306657 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzfp8\" (UniqueName: \"kubernetes.io/projected/3324d5df-e2ef-46ab-89b2-3b64126732b9-kube-api-access-lzfp8\") pod \"maas-api-revoke-sub-6d9f4c53-htbbm\" (UID: \"3324d5df-e2ef-46ab-89b2-3b64126732b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9f4c53-htbbm" Oct 30 13:29:38.323112 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:38.323092 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9f4c53-htbbm" Oct 30 13:29:38.635148 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:38.635118 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d9f4c53-htbbm"] Oct 30 13:29:38.636678 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:29:38.636647 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3324d5df_e2ef_46ab_89b2_3b64126732b9.slice/crio-2941f59cca22bb58675f75e715252510e743c550ed553059c5d7151bacf8acc5 WatchSource:0}: Error finding container 2941f59cca22bb58675f75e715252510e743c550ed553059c5d7151bacf8acc5: Status 404 returned error can't find the container with id 2941f59cca22bb58675f75e715252510e743c550ed553059c5d7151bacf8acc5 Oct 30 13:29:39.374054 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:39.373957 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9f4c53-htbbm" event={"ID":"3324d5df-e2ef-46ab-89b2-3b64126732b9","Type":"ContainerStarted","Data":"0b95ea9f858bb956f5c08777648f57d9ae4fc09fd98325dd5f3d786573f01475"} Oct 30 13:29:39.374054 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:39.374023 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9f4c53-htbbm" event={"ID":"3324d5df-e2ef-46ab-89b2-3b64126732b9","Type":"ContainerStarted","Data":"2941f59cca22bb58675f75e715252510e743c550ed553059c5d7151bacf8acc5"} Oct 30 13:29:39.374322 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:39.374111 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx"] Oct 30 13:29:39.378317 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:39.378300 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" Oct 30 13:29:39.405430 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:39.405403 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx"] Oct 30 13:29:39.419870 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:39.419823 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9f4c53-htbbm" podStartSLOduration=2.419811707 podStartE2EDuration="2.419811707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:39.416425196 +0000 UTC m=+2678.019923643" watchObservedRunningTime="2026-10-30 13:29:39.419811707 +0000 UTC m=+2678.023310152" Oct 30 13:29:39.499880 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:39.499846 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/defbc903-546b-4dfd-a284-50e517429e6f-istio-podinfo\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" Oct 30 13:29:39.499880 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:39.499886 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/defbc903-546b-4dfd-a284-50e517429e6f-istiod-ca-cert\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" Oct 30 13:29:39.500127 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:39.500108 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/defbc903-546b-4dfd-a284-50e517429e6f-workload-socket\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" Oct 30 13:29:39.500184 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:39.500150 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/defbc903-546b-4dfd-a284-50e517429e6f-istio-token\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" Oct 30 13:29:39.500276 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:39.500237 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/defbc903-546b-4dfd-a284-50e517429e6f-credential-socket\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" Oct 30 13:29:39.500332 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:39.500317 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/defbc903-546b-4dfd-a284-50e517429e6f-workload-certs\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" Oct 30 13:29:39.500388 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:39.500352 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/defbc903-546b-4dfd-a284-50e517429e6f-istio-envoy\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" Oct 30 13:29:39.500468 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:39.500447 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/defbc903-546b-4dfd-a284-50e517429e6f-istio-data\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" Oct 30 13:29:39.500688 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:39.500662 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbdtc\" (UniqueName: \"kubernetes.io/projected/defbc903-546b-4dfd-a284-50e517429e6f-kube-api-access-vbdtc\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" Oct 30 13:29:39.601929 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:39.601901 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/defbc903-546b-4dfd-a284-50e517429e6f-istiod-ca-cert\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" Oct 30 13:29:39.602058 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:39.601953 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/defbc903-546b-4dfd-a284-50e517429e6f-workload-socket\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" Oct 30 13:29:39.602058 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:39.601992 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/defbc903-546b-4dfd-a284-50e517429e6f-istio-token\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" Oct 30 13:29:39.602058 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:39.602031 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/defbc903-546b-4dfd-a284-50e517429e6f-credential-socket\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" Oct 30 13:29:39.602226 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:39.602062 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/defbc903-546b-4dfd-a284-50e517429e6f-workload-certs\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" Oct 30 13:29:39.602226 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:39.602096 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/defbc903-546b-4dfd-a284-50e517429e6f-istio-envoy\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" Oct 30 13:29:39.602226 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:39.602154 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/defbc903-546b-4dfd-a284-50e517429e6f-istio-data\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" Oct 30 13:29:39.602419 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:39.602254 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbdtc\" (UniqueName: \"kubernetes.io/projected/defbc903-546b-4dfd-a284-50e517429e6f-kube-api-access-vbdtc\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" Oct 30 13:29:39.602480 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:39.602435 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/defbc903-546b-4dfd-a284-50e517429e6f-istio-podinfo\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" Oct 30 13:29:39.602518 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:39.602482 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/defbc903-546b-4dfd-a284-50e517429e6f-workload-socket\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" Oct 30 13:29:39.602790 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:39.602764 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/defbc903-546b-4dfd-a284-50e517429e6f-credential-socket\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" Oct 30 13:29:39.602987 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:39.602828 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/defbc903-546b-4dfd-a284-50e517429e6f-istiod-ca-cert\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" Oct 30 13:29:39.602987 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:39.602845 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/defbc903-546b-4dfd-a284-50e517429e6f-workload-certs\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" Oct 30 13:29:39.602987 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:39.602861 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/defbc903-546b-4dfd-a284-50e517429e6f-istio-data\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" Oct 30 13:29:39.604536 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:39.604511 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/defbc903-546b-4dfd-a284-50e517429e6f-istio-envoy\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" Oct 30 13:29:39.605061 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:39.605045 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/defbc903-546b-4dfd-a284-50e517429e6f-istio-podinfo\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" Oct 30 13:29:39.626946 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:39.626890 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/defbc903-546b-4dfd-a284-50e517429e6f-istio-token\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" Oct 30 13:29:39.627058 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:39.627021 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbdtc\" (UniqueName: \"kubernetes.io/projected/defbc903-546b-4dfd-a284-50e517429e6f-kube-api-access-vbdtc\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" Oct 30 13:29:39.692440 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:39.692412 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" Oct 30 13:29:40.108367 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:29:40.108331 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddefbc903_546b_4dfd_a284_50e517429e6f.slice/crio-64f03daa42576202fc59842299485a3b3f775a9be9efc267fef1a07565433440 WatchSource:0}: Error finding container 64f03daa42576202fc59842299485a3b3f775a9be9efc267fef1a07565433440: Status 404 returned error can't find the container with id 64f03daa42576202fc59842299485a3b3f775a9be9efc267fef1a07565433440 Oct 30 13:29:40.109887 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:40.109851 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx"] Oct 30 13:29:40.111761 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:40.111727 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:29:40.111850 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:40.111793 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:29:40.111850 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:40.111822 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:29:40.383191 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:40.383092 4931 generic.go:356] "Generic (PLEG): container finished" podID="3324d5df-e2ef-46ab-89b2-3b64126732b9" containerID="0b95ea9f858bb956f5c08777648f57d9ae4fc09fd98325dd5f3d786573f01475" exitCode=0 Oct 30 13:29:40.383402 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:40.383177 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9f4c53-htbbm" event={"ID":"3324d5df-e2ef-46ab-89b2-3b64126732b9","Type":"ContainerDied","Data":"0b95ea9f858bb956f5c08777648f57d9ae4fc09fd98325dd5f3d786573f01475"} Oct 30 13:29:40.385069 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:40.385045 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" event={"ID":"defbc903-546b-4dfd-a284-50e517429e6f","Type":"ContainerStarted","Data":"eb83281d54ca11df15a7cede8ac4475182b4a01550a5bbee6d7d379e49c02242"} Oct 30 13:29:40.385188 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:40.385091 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" event={"ID":"defbc903-546b-4dfd-a284-50e517429e6f","Type":"ContainerStarted","Data":"64f03daa42576202fc59842299485a3b3f775a9be9efc267fef1a07565433440"} Oct 30 13:29:40.477648 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:40.477593 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" podStartSLOduration=1.4775791950000001 podStartE2EDuration="1.477579195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:40.473539958 +0000 UTC m=+2679.077038406" watchObservedRunningTime="2026-10-30 13:29:40.477579195 +0000 UTC m=+2679.081077640" Oct 30 13:29:40.694434 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:40.694392 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" Oct 30 13:29:41.624406 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:41.624377 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9f4c53-htbbm" Oct 30 13:29:41.663325 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:41.663196 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3324d5df-e2ef-46ab-89b2-3b64126732b9-kube-api-access-lzfp8\" (UniqueName: \"kubernetes.io/projected/3324d5df-e2ef-46ab-89b2-3b64126732b9-kube-api-access-lzfp8\") pod \"3324d5df-e2ef-46ab-89b2-3b64126732b9\" (UID: \"3324d5df-e2ef-46ab-89b2-3b64126732b9\") " Oct 30 13:29:41.663325 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:41.663232 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3324d5df-e2ef-46ab-89b2-3b64126732b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3324d5df-e2ef-46ab-89b2-3b64126732b9-maas-api-xks-ca\") pod \"3324d5df-e2ef-46ab-89b2-3b64126732b9\" (UID: \"3324d5df-e2ef-46ab-89b2-3b64126732b9\") " Oct 30 13:29:41.663325 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:41.663306 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3324d5df-e2ef-46ab-89b2-3b64126732b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3324d5df-e2ef-46ab-89b2-3b64126732b9-maas-api-openshift-ca\") pod \"3324d5df-e2ef-46ab-89b2-3b64126732b9\" (UID: \"3324d5df-e2ef-46ab-89b2-3b64126732b9\") " Oct 30 13:29:41.663824 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:41.663777 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3324d5df-e2ef-46ab-89b2-3b64126732b9-maas-api-openshift-ca" pod "3324d5df-e2ef-46ab-89b2-3b64126732b9" (UID: "3324d5df-e2ef-46ab-89b2-3b64126732b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:29:41.665477 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:41.665451 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3324d5df-e2ef-46ab-89b2-3b64126732b9-kube-api-access-lzfp8" pod "3324d5df-e2ef-46ab-89b2-3b64126732b9" (UID: "3324d5df-e2ef-46ab-89b2-3b64126732b9"). InnerVolumeSpecName "kube-api-access-lzfp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:41.665595 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:41.665483 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3324d5df-e2ef-46ab-89b2-3b64126732b9-maas-api-xks-ca" pod "3324d5df-e2ef-46ab-89b2-3b64126732b9" (UID: "3324d5df-e2ef-46ab-89b2-3b64126732b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:29:41.695202 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:41.695169 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 13:29:41.695602 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:41.695230 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" podUID="defbc903-546b-4dfd-a284-50e517429e6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 13:29:41.764893 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:41.764853 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzfp8\" (UniqueName: \"kubernetes.io/projected/3324d5df-e2ef-46ab-89b2-3b64126732b9-kube-api-access-lzfp8\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:29:41.764893 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:41.764886 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3324d5df-e2ef-46ab-89b2-3b64126732b9-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:29:41.764893 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:41.764896 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3324d5df-e2ef-46ab-89b2-3b64126732b9-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:29:42.402045 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:42.402017 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9f4c53-htbbm" Oct 30 13:29:42.402226 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:42.402059 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9f4c53-htbbm" event={"ID":"3324d5df-e2ef-46ab-89b2-3b64126732b9","Type":"ContainerDied","Data":"2941f59cca22bb58675f75e715252510e743c550ed553059c5d7151bacf8acc5"} Oct 30 13:29:42.402226 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:42.402110 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2941f59cca22bb58675f75e715252510e743c550ed553059c5d7151bacf8acc5" Oct 30 13:29:42.693164 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:42.693133 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 13:29:42.693355 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:42.693188 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" podUID="defbc903-546b-4dfd-a284-50e517429e6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 13:29:43.693578 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:43.693539 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 13:29:43.694017 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:43.693598 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" podUID="defbc903-546b-4dfd-a284-50e517429e6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 13:29:44.693016 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:44.692979 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 13:29:44.693211 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:44.693038 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" podUID="defbc903-546b-4dfd-a284-50e517429e6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 13:29:45.692723 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:45.692687 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 13:29:45.693137 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:45.692749 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" podUID="defbc903-546b-4dfd-a284-50e517429e6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 13:29:45.695495 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:45.695468 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r"] Oct 30 13:29:45.696306 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:45.696289 4931 cpu_manager.go:414] "containerMap: removing container" podUID="3324d5df-e2ef-46ab-89b2-3b64126732b9" containerName="revoke-keys" Oct 30 13:29:45.696356 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:45.696309 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3324d5df-e2ef-46ab-89b2-3b64126732b9" containerName="revoke-keys" Oct 30 13:29:45.696476 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:45.696463 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="3324d5df-e2ef-46ab-89b2-3b64126732b9" containerName="revoke-keys" Oct 30 13:29:45.700684 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:45.700664 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r" Oct 30 13:29:45.749292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:45.747292 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-acd125\"/\"default-dockercfg-hfjxl\"" Oct 30 13:29:45.749292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:45.747445 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-acd125\"/\"rate-test-model-acd125-kserve-self-signed-certs\"" Oct 30 13:29:45.749292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:45.747541 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-acd125\"/\"kube-root-ca.crt\"" Oct 30 13:29:45.749292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:45.747609 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-acd125\"/\"openshift-service-ca.crt\"" Oct 30 13:29:45.782584 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:45.782538 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r"] Oct 30 13:29:45.819851 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:45.819801 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f88311-b26d-4433-804e-5a2f05180694-model-cache\") pod \"rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r\" (UID: \"f5f88311-b26d-4433-804e-5a2f05180694\") " pod="ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r" Oct 30 13:29:45.819851 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:45.819856 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5f88311-b26d-4433-804e-5a2f05180694-dshm\") pod \"rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r\" (UID: \"f5f88311-b26d-4433-804e-5a2f05180694\") " pod="ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r" Oct 30 13:29:45.820102 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:45.820074 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5f88311-b26d-4433-804e-5a2f05180694-tmp-dir\") pod \"rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r\" (UID: \"f5f88311-b26d-4433-804e-5a2f05180694\") " pod="ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r" Oct 30 13:29:45.820211 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:45.820193 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vqbq\" (UniqueName: \"kubernetes.io/projected/f5f88311-b26d-4433-804e-5a2f05180694-kube-api-access-2vqbq\") pod \"rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r\" (UID: \"f5f88311-b26d-4433-804e-5a2f05180694\") " pod="ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r" Oct 30 13:29:45.820290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:45.820258 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f88311-b26d-4433-804e-5a2f05180694-tls-certs\") pod \"rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r\" (UID: \"f5f88311-b26d-4433-804e-5a2f05180694\") " pod="ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r" Oct 30 13:29:45.820369 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:45.820356 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5f88311-b26d-4433-804e-5a2f05180694-home\") pod \"rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r\" (UID: \"f5f88311-b26d-4433-804e-5a2f05180694\") " pod="ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r" Oct 30 13:29:45.921709 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:45.921664 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5f88311-b26d-4433-804e-5a2f05180694-home\") pod \"rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r\" (UID: \"f5f88311-b26d-4433-804e-5a2f05180694\") " pod="ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r" Oct 30 13:29:45.921954 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:45.921746 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f88311-b26d-4433-804e-5a2f05180694-model-cache\") pod \"rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r\" (UID: \"f5f88311-b26d-4433-804e-5a2f05180694\") " pod="ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r" Oct 30 13:29:45.921954 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:45.921775 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5f88311-b26d-4433-804e-5a2f05180694-dshm\") pod \"rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r\" (UID: \"f5f88311-b26d-4433-804e-5a2f05180694\") " pod="ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r" Oct 30 13:29:45.921954 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:45.921904 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5f88311-b26d-4433-804e-5a2f05180694-tmp-dir\") pod \"rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r\" (UID: \"f5f88311-b26d-4433-804e-5a2f05180694\") " pod="ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r" Oct 30 13:29:45.922138 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:45.922009 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vqbq\" (UniqueName: \"kubernetes.io/projected/f5f88311-b26d-4433-804e-5a2f05180694-kube-api-access-2vqbq\") pod \"rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r\" (UID: \"f5f88311-b26d-4433-804e-5a2f05180694\") " pod="ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r" Oct 30 13:29:45.922138 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:45.922071 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f88311-b26d-4433-804e-5a2f05180694-tls-certs\") pod \"rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r\" (UID: \"f5f88311-b26d-4433-804e-5a2f05180694\") " pod="ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r" Oct 30 13:29:45.922247 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:45.922218 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5f88311-b26d-4433-804e-5a2f05180694-home\") pod \"rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r\" (UID: \"f5f88311-b26d-4433-804e-5a2f05180694\") " pod="ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r" Oct 30 13:29:45.922382 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:45.922354 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5f88311-b26d-4433-804e-5a2f05180694-tmp-dir\") pod \"rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r\" (UID: \"f5f88311-b26d-4433-804e-5a2f05180694\") " pod="ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r" Oct 30 13:29:45.922560 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:45.922536 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f88311-b26d-4433-804e-5a2f05180694-model-cache\") pod \"rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r\" (UID: \"f5f88311-b26d-4433-804e-5a2f05180694\") " pod="ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r" Oct 30 13:29:45.924432 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:45.924406 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5f88311-b26d-4433-804e-5a2f05180694-dshm\") pod \"rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r\" (UID: \"f5f88311-b26d-4433-804e-5a2f05180694\") " pod="ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r" Oct 30 13:29:45.925058 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:45.925034 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f88311-b26d-4433-804e-5a2f05180694-tls-certs\") pod \"rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r\" (UID: \"f5f88311-b26d-4433-804e-5a2f05180694\") " pod="ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r" Oct 30 13:29:45.963152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:45.963073 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vqbq\" (UniqueName: \"kubernetes.io/projected/f5f88311-b26d-4433-804e-5a2f05180694-kube-api-access-2vqbq\") pod \"rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r\" (UID: \"f5f88311-b26d-4433-804e-5a2f05180694\") " pod="ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r" Oct 30 13:29:46.018291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:46.014591 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r" Oct 30 13:29:46.463178 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:29:46.463142 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5f88311_b26d_4433_804e_5a2f05180694.slice/crio-0e90a62babe87371c64017bdcba8d74ad00ddc3f4c22ef1c9ab9dee3ff89a4e4 WatchSource:0}: Error finding container 0e90a62babe87371c64017bdcba8d74ad00ddc3f4c22ef1c9ab9dee3ff89a4e4: Status 404 returned error can't find the container with id 0e90a62babe87371c64017bdcba8d74ad00ddc3f4c22ef1c9ab9dee3ff89a4e4 Oct 30 13:29:46.463798 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:46.463760 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r"] Oct 30 13:29:46.697718 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:46.697685 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" Oct 30 13:29:46.698195 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:46.698176 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" Oct 30 13:29:46.698879 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:46.698857 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" Oct 30 13:29:47.449133 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:47.449089 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r" event={"ID":"f5f88311-b26d-4433-804e-5a2f05180694","Type":"ContainerStarted","Data":"5afffceba04b175616d91d7137f3617f84b1fc774f13329650b5573b7cdbee0d"} Oct 30 13:29:47.449359 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:47.449141 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r" event={"ID":"f5f88311-b26d-4433-804e-5a2f05180694","Type":"ContainerStarted","Data":"0e90a62babe87371c64017bdcba8d74ad00ddc3f4c22ef1c9ab9dee3ff89a4e4"} Oct 30 13:29:47.504311 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:47.498485 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r" podStartSLOduration=2.498469252 podStartE2EDuration="2.498469252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:47.494231806 +0000 UTC m=+2686.097730252" watchObservedRunningTime="2026-10-30 13:29:47.498469252 +0000 UTC m=+2686.101967699" Oct 30 13:29:48.015175 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:48.015141 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r" Oct 30 13:29:48.902654 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:48.902620 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc79a401-n6895"] Oct 30 13:29:48.906904 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:48.906883 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc79a401-n6895" Oct 30 13:29:48.937105 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:48.937074 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:29:48.937289 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:48.937135 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:29:48.941434 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:48.941408 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc79a401-n6895"] Oct 30 13:29:48.968622 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:48.968567 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7ktq\" (UniqueName: \"kubernetes.io/projected/f22bdb08-5157-454d-a5a5-b94f96d135fb-kube-api-access-l7ktq\") pod \"maas-api-revoke-sub-cc79a401-n6895\" (UID: \"f22bdb08-5157-454d-a5a5-b94f96d135fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc79a401-n6895" Oct 30 13:29:48.968822 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:48.968644 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f22bdb08-5157-454d-a5a5-b94f96d135fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc79a401-n6895\" (UID: \"f22bdb08-5157-454d-a5a5-b94f96d135fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc79a401-n6895" Oct 30 13:29:48.969038 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:48.969013 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f22bdb08-5157-454d-a5a5-b94f96d135fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc79a401-n6895\" (UID: \"f22bdb08-5157-454d-a5a5-b94f96d135fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc79a401-n6895" Oct 30 13:29:49.071047 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:49.071012 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7ktq\" (UniqueName: \"kubernetes.io/projected/f22bdb08-5157-454d-a5a5-b94f96d135fb-kube-api-access-l7ktq\") pod \"maas-api-revoke-sub-cc79a401-n6895\" (UID: \"f22bdb08-5157-454d-a5a5-b94f96d135fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc79a401-n6895" Oct 30 13:29:49.071047 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:49.071059 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f22bdb08-5157-454d-a5a5-b94f96d135fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc79a401-n6895\" (UID: \"f22bdb08-5157-454d-a5a5-b94f96d135fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc79a401-n6895" Oct 30 13:29:49.071612 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:49.071211 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f22bdb08-5157-454d-a5a5-b94f96d135fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc79a401-n6895\" (UID: \"f22bdb08-5157-454d-a5a5-b94f96d135fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc79a401-n6895" Oct 30 13:29:49.071913 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:49.071885 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f22bdb08-5157-454d-a5a5-b94f96d135fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc79a401-n6895\" (UID: \"f22bdb08-5157-454d-a5a5-b94f96d135fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc79a401-n6895" Oct 30 13:29:49.073543 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:49.073522 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f22bdb08-5157-454d-a5a5-b94f96d135fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc79a401-n6895\" (UID: \"f22bdb08-5157-454d-a5a5-b94f96d135fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc79a401-n6895" Oct 30 13:29:49.096201 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:49.096163 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7ktq\" (UniqueName: \"kubernetes.io/projected/f22bdb08-5157-454d-a5a5-b94f96d135fb-kube-api-access-l7ktq\") pod \"maas-api-revoke-sub-cc79a401-n6895\" (UID: \"f22bdb08-5157-454d-a5a5-b94f96d135fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc79a401-n6895" Oct 30 13:29:49.221529 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:49.221495 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc79a401-n6895" Oct 30 13:29:49.539019 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:49.538979 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc79a401-n6895"] Oct 30 13:29:49.539331 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:29:49.539305 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf22bdb08_5157_454d_a5a5_b94f96d135fb.slice/crio-ab5d53c575556ff3d631412a7cb64e45e7110648abee83a99c563437a8eb8cf0 WatchSource:0}: Error finding container ab5d53c575556ff3d631412a7cb64e45e7110648abee83a99c563437a8eb8cf0: Status 404 returned error can't find the container with id ab5d53c575556ff3d631412a7cb64e45e7110648abee83a99c563437a8eb8cf0 Oct 30 13:29:50.474195 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:50.474168 4931 generic.go:356] "Generic (PLEG): container finished" podID="f22bdb08-5157-454d-a5a5-b94f96d135fb" containerID="5a72eeb3a0ae9faec2e0c1f0f4cd7d1418e3c46d3dac3e821514e5abdb04c99b" exitCode=0 Oct 30 13:29:50.474571 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:50.474251 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc79a401-n6895" event={"ID":"f22bdb08-5157-454d-a5a5-b94f96d135fb","Type":"ContainerDied","Data":"5a72eeb3a0ae9faec2e0c1f0f4cd7d1418e3c46d3dac3e821514e5abdb04c99b"} Oct 30 13:29:50.474571 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:50.474336 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc79a401-n6895" event={"ID":"f22bdb08-5157-454d-a5a5-b94f96d135fb","Type":"ContainerStarted","Data":"ab5d53c575556ff3d631412a7cb64e45e7110648abee83a99c563437a8eb8cf0"} Oct 30 13:29:51.644159 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:51.644136 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc79a401-n6895" Oct 30 13:29:51.715191 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:51.715147 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f22bdb08-5157-454d-a5a5-b94f96d135fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f22bdb08-5157-454d-a5a5-b94f96d135fb-maas-api-xks-ca\") pod \"f22bdb08-5157-454d-a5a5-b94f96d135fb\" (UID: \"f22bdb08-5157-454d-a5a5-b94f96d135fb\") " Oct 30 13:29:51.715404 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:51.715246 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f22bdb08-5157-454d-a5a5-b94f96d135fb-kube-api-access-l7ktq\" (UniqueName: \"kubernetes.io/projected/f22bdb08-5157-454d-a5a5-b94f96d135fb-kube-api-access-l7ktq\") pod \"f22bdb08-5157-454d-a5a5-b94f96d135fb\" (UID: \"f22bdb08-5157-454d-a5a5-b94f96d135fb\") " Oct 30 13:29:51.715404 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:51.715393 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f22bdb08-5157-454d-a5a5-b94f96d135fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f22bdb08-5157-454d-a5a5-b94f96d135fb-maas-api-openshift-ca\") pod \"f22bdb08-5157-454d-a5a5-b94f96d135fb\" (UID: \"f22bdb08-5157-454d-a5a5-b94f96d135fb\") " Oct 30 13:29:51.719722 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:51.719683 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f22bdb08-5157-454d-a5a5-b94f96d135fb-maas-api-openshift-ca" pod "f22bdb08-5157-454d-a5a5-b94f96d135fb" (UID: "f22bdb08-5157-454d-a5a5-b94f96d135fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:29:51.720097 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:51.720068 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f22bdb08-5157-454d-a5a5-b94f96d135fb-maas-api-xks-ca" pod "f22bdb08-5157-454d-a5a5-b94f96d135fb" (UID: "f22bdb08-5157-454d-a5a5-b94f96d135fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:29:51.727819 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:51.727788 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f22bdb08-5157-454d-a5a5-b94f96d135fb-kube-api-access-l7ktq" pod "f22bdb08-5157-454d-a5a5-b94f96d135fb" (UID: "f22bdb08-5157-454d-a5a5-b94f96d135fb"). InnerVolumeSpecName "kube-api-access-l7ktq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:51.816902 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:51.816866 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f22bdb08-5157-454d-a5a5-b94f96d135fb-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:29:51.816902 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:51.816895 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7ktq\" (UniqueName: \"kubernetes.io/projected/f22bdb08-5157-454d-a5a5-b94f96d135fb-kube-api-access-l7ktq\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:29:51.816902 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:51.816905 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f22bdb08-5157-454d-a5a5-b94f96d135fb-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:29:52.490741 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:52.490710 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc79a401-n6895" Oct 30 13:29:52.490910 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:52.490736 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc79a401-n6895" event={"ID":"f22bdb08-5157-454d-a5a5-b94f96d135fb","Type":"ContainerDied","Data":"ab5d53c575556ff3d631412a7cb64e45e7110648abee83a99c563437a8eb8cf0"} Oct 30 13:29:52.490910 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:52.490785 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab5d53c575556ff3d631412a7cb64e45e7110648abee83a99c563437a8eb8cf0" Oct 30 13:29:56.015354 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:56.015315 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r" Oct 30 13:29:56.028750 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:56.028722 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r" Oct 30 13:29:56.533416 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:56.533381 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r" Oct 30 13:29:59.603468 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:59.603433 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5420598-fmxqg"] Oct 30 13:29:59.604227 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:59.604204 4931 cpu_manager.go:414] "containerMap: removing container" podUID="f22bdb08-5157-454d-a5a5-b94f96d135fb" containerName="revoke-keys" Oct 30 13:29:59.604362 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:59.604229 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f22bdb08-5157-454d-a5a5-b94f96d135fb" containerName="revoke-keys" Oct 30 13:29:59.604428 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:59.604415 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f22bdb08-5157-454d-a5a5-b94f96d135fb" containerName="revoke-keys" Oct 30 13:29:59.608256 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:59.608233 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5420598-fmxqg" Oct 30 13:29:59.630307 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:59.630281 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5420598-fmxqg"] Oct 30 13:29:59.637513 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:59.637488 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:29:59.637833 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:59.637806 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:29:59.729928 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:59.729871 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk4gp\" (UniqueName: \"kubernetes.io/projected/b0716cec-3e99-4ef4-ad57-61dc8bfc9c75-kube-api-access-jk4gp\") pod \"maas-api-revoke-sub-c5420598-fmxqg\" (UID: \"b0716cec-3e99-4ef4-ad57-61dc8bfc9c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5420598-fmxqg" Oct 30 13:29:59.730107 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:59.730012 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0716cec-3e99-4ef4-ad57-61dc8bfc9c75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5420598-fmxqg\" (UID: \"b0716cec-3e99-4ef4-ad57-61dc8bfc9c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5420598-fmxqg" Oct 30 13:29:59.730107 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:59.730092 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0716cec-3e99-4ef4-ad57-61dc8bfc9c75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5420598-fmxqg\" (UID: \"b0716cec-3e99-4ef4-ad57-61dc8bfc9c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5420598-fmxqg" Oct 30 13:29:59.831387 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:59.831354 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk4gp\" (UniqueName: \"kubernetes.io/projected/b0716cec-3e99-4ef4-ad57-61dc8bfc9c75-kube-api-access-jk4gp\") pod \"maas-api-revoke-sub-c5420598-fmxqg\" (UID: \"b0716cec-3e99-4ef4-ad57-61dc8bfc9c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5420598-fmxqg" Oct 30 13:29:59.831597 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:59.831467 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0716cec-3e99-4ef4-ad57-61dc8bfc9c75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5420598-fmxqg\" (UID: \"b0716cec-3e99-4ef4-ad57-61dc8bfc9c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5420598-fmxqg" Oct 30 13:29:59.831597 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:59.831534 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0716cec-3e99-4ef4-ad57-61dc8bfc9c75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5420598-fmxqg\" (UID: \"b0716cec-3e99-4ef4-ad57-61dc8bfc9c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5420598-fmxqg" Oct 30 13:29:59.832503 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:59.832470 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0716cec-3e99-4ef4-ad57-61dc8bfc9c75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5420598-fmxqg\" (UID: \"b0716cec-3e99-4ef4-ad57-61dc8bfc9c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5420598-fmxqg" Oct 30 13:29:59.834074 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:59.834046 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0716cec-3e99-4ef4-ad57-61dc8bfc9c75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5420598-fmxqg\" (UID: \"b0716cec-3e99-4ef4-ad57-61dc8bfc9c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5420598-fmxqg" Oct 30 13:29:59.856187 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:59.856122 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk4gp\" (UniqueName: \"kubernetes.io/projected/b0716cec-3e99-4ef4-ad57-61dc8bfc9c75-kube-api-access-jk4gp\") pod \"maas-api-revoke-sub-c5420598-fmxqg\" (UID: \"b0716cec-3e99-4ef4-ad57-61dc8bfc9c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5420598-fmxqg" Oct 30 13:29:59.923402 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:29:59.923358 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5420598-fmxqg" Oct 30 13:30:00.174055 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.172712 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t"] Oct 30 13:30:00.179300 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.179251 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t" Oct 30 13:30:00.210955 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.210741 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb"] Oct 30 13:30:00.231634 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.230562 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd"] Oct 30 13:30:00.231634 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.230868 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb" Oct 30 13:30:00.241687 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.241555 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq"] Oct 30 13:30:00.242155 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.241648 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd" Oct 30 13:30:00.251645 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.251623 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t"] Oct 30 13:30:00.251907 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.251889 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg"] Oct 30 13:30:00.253380 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.253359 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq" Oct 30 13:30:00.300305 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.293035 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z"] Oct 30 13:30:00.352874 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.327075 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmddv\" (UniqueName: \"kubernetes.io/projected/122219fe-8078-42d9-b4ed-7c0333b9013c-kube-api-access-lmddv\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t\" (UID: \"122219fe-8078-42d9-b4ed-7c0333b9013c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t" Oct 30 13:30:00.352874 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.327153 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32712e88-00fd-4f97-882a-a633da6a16df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq\" (UID: \"32712e88-00fd-4f97-882a-a633da6a16df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq" Oct 30 13:30:00.352874 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.327316 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52d65cd3-640f-46e7-90e9-534c3c990930-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb\" (UID: \"52d65cd3-640f-46e7-90e9-534c3c990930\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb" Oct 30 13:30:00.352874 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.327410 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4pbp\" (UniqueName: \"kubernetes.io/projected/32712e88-00fd-4f97-882a-a633da6a16df-kube-api-access-b4pbp\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq\" (UID: \"32712e88-00fd-4f97-882a-a633da6a16df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq" Oct 30 13:30:00.352874 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.327481 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z57n7\" (UniqueName: \"kubernetes.io/projected/52d65cd3-640f-46e7-90e9-534c3c990930-kube-api-access-z57n7\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb\" (UID: \"52d65cd3-640f-46e7-90e9-534c3c990930\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb" Oct 30 13:30:00.352874 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.327566 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22fwv\" (UniqueName: \"kubernetes.io/projected/ae4551d9-3f2a-4278-b567-243121f8e1ff-kube-api-access-22fwv\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd\" (UID: \"ae4551d9-3f2a-4278-b567-243121f8e1ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd" Oct 30 13:30:00.352874 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.327630 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/122219fe-8078-42d9-b4ed-7c0333b9013c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t\" (UID: \"122219fe-8078-42d9-b4ed-7c0333b9013c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t" Oct 30 13:30:00.352874 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.327667 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae4551d9-3f2a-4278-b567-243121f8e1ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd\" (UID: \"ae4551d9-3f2a-4278-b567-243121f8e1ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd" Oct 30 13:30:00.352874 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.337008 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg" Oct 30 13:30:00.435333 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.434575 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmddv\" (UniqueName: \"kubernetes.io/projected/122219fe-8078-42d9-b4ed-7c0333b9013c-kube-api-access-lmddv\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t\" (UID: \"122219fe-8078-42d9-b4ed-7c0333b9013c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t" Oct 30 13:30:00.435333 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.434635 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32712e88-00fd-4f97-882a-a633da6a16df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq\" (UID: \"32712e88-00fd-4f97-882a-a633da6a16df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq" Oct 30 13:30:00.436361 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.435600 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32712e88-00fd-4f97-882a-a633da6a16df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq\" (UID: \"32712e88-00fd-4f97-882a-a633da6a16df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq" Oct 30 13:30:00.436361 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.435657 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52d65cd3-640f-46e7-90e9-534c3c990930-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb\" (UID: \"52d65cd3-640f-46e7-90e9-534c3c990930\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb" Oct 30 13:30:00.436361 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.435673 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb"] Oct 30 13:30:00.436361 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.435736 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4pbp\" (UniqueName: \"kubernetes.io/projected/32712e88-00fd-4f97-882a-a633da6a16df-kube-api-access-b4pbp\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq\" (UID: \"32712e88-00fd-4f97-882a-a633da6a16df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq" Oct 30 13:30:00.436361 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.435784 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z" Oct 30 13:30:00.436361 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.435805 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z57n7\" (UniqueName: \"kubernetes.io/projected/52d65cd3-640f-46e7-90e9-534c3c990930-kube-api-access-z57n7\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb\" (UID: \"52d65cd3-640f-46e7-90e9-534c3c990930\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb" Oct 30 13:30:00.436361 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.435854 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f88ff2-f7d4-4207-b2d1-a49a6f052a2c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg\" (UID: \"65f88ff2-f7d4-4207-b2d1-a49a6f052a2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg" Oct 30 13:30:00.436361 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.435895 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22fwv\" (UniqueName: \"kubernetes.io/projected/ae4551d9-3f2a-4278-b567-243121f8e1ff-kube-api-access-22fwv\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd\" (UID: \"ae4551d9-3f2a-4278-b567-243121f8e1ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd" Oct 30 13:30:00.436361 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.435937 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxm7r\" (UniqueName: \"kubernetes.io/projected/65f88ff2-f7d4-4207-b2d1-a49a6f052a2c-kube-api-access-hxm7r\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg\" (UID: \"65f88ff2-f7d4-4207-b2d1-a49a6f052a2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg" Oct 30 13:30:00.436361 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.435971 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/122219fe-8078-42d9-b4ed-7c0333b9013c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t\" (UID: \"122219fe-8078-42d9-b4ed-7c0333b9013c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t" Oct 30 13:30:00.436361 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.436008 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae4551d9-3f2a-4278-b567-243121f8e1ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd\" (UID: \"ae4551d9-3f2a-4278-b567-243121f8e1ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd" Oct 30 13:30:00.438035 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.437878 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52d65cd3-640f-46e7-90e9-534c3c990930-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb\" (UID: \"52d65cd3-640f-46e7-90e9-534c3c990930\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb" Oct 30 13:30:00.438035 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.437991 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae4551d9-3f2a-4278-b567-243121f8e1ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd\" (UID: \"ae4551d9-3f2a-4278-b567-243121f8e1ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd" Oct 30 13:30:00.439687 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.439660 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/122219fe-8078-42d9-b4ed-7c0333b9013c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t\" (UID: \"122219fe-8078-42d9-b4ed-7c0333b9013c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t" Oct 30 13:30:00.458880 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.458855 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq"] Oct 30 13:30:00.465789 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.465752 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd"] Oct 30 13:30:00.466315 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.466245 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4pbp\" (UniqueName: \"kubernetes.io/projected/32712e88-00fd-4f97-882a-a633da6a16df-kube-api-access-b4pbp\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq\" (UID: \"32712e88-00fd-4f97-882a-a633da6a16df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq" Oct 30 13:30:00.469171 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.469094 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmddv\" (UniqueName: \"kubernetes.io/projected/122219fe-8078-42d9-b4ed-7c0333b9013c-kube-api-access-lmddv\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t\" (UID: \"122219fe-8078-42d9-b4ed-7c0333b9013c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t" Oct 30 13:30:00.470397 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.470373 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z57n7\" (UniqueName: \"kubernetes.io/projected/52d65cd3-640f-46e7-90e9-534c3c990930-kube-api-access-z57n7\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb\" (UID: \"52d65cd3-640f-46e7-90e9-534c3c990930\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb" Oct 30 13:30:00.470847 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.470828 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq" Oct 30 13:30:00.471400 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.471380 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22fwv\" (UniqueName: \"kubernetes.io/projected/ae4551d9-3f2a-4278-b567-243121f8e1ff-kube-api-access-22fwv\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd\" (UID: \"ae4551d9-3f2a-4278-b567-243121f8e1ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd" Oct 30 13:30:00.472401 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.472379 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889450-bnskt"] Oct 30 13:30:00.479104 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.478009 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg"] Oct 30 13:30:00.479104 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.478115 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-bnskt" Oct 30 13:30:00.482095 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.482071 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z"] Oct 30 13:30:00.493109 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.493082 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj"] Oct 30 13:30:00.536290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.525310 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t" Oct 30 13:30:00.536290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.528724 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889450-7dzx8"] Oct 30 13:30:00.536290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.535937 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj" Oct 30 13:30:00.536758 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.536462 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889450-7dzx8" Oct 30 13:30:00.566357 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.546890 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f88ff2-f7d4-4207-b2d1-a49a6f052a2c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg\" (UID: \"65f88ff2-f7d4-4207-b2d1-a49a6f052a2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg" Oct 30 13:30:00.566357 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.546996 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxm7r\" (UniqueName: \"kubernetes.io/projected/65f88ff2-f7d4-4207-b2d1-a49a6f052a2c-kube-api-access-hxm7r\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg\" (UID: \"65f88ff2-f7d4-4207-b2d1-a49a6f052a2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg" Oct 30 13:30:00.566357 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.547030 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5592ca1-ccf3-4297-bc42-ea5aa0cca97a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z\" (UID: \"a5592ca1-ccf3-4297-bc42-ea5aa0cca97a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z" Oct 30 13:30:00.566357 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.547075 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6f6b\" (UniqueName: \"kubernetes.io/projected/a5592ca1-ccf3-4297-bc42-ea5aa0cca97a-kube-api-access-v6f6b\") pod \"maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z\" (UID: \"a5592ca1-ccf3-4297-bc42-ea5aa0cca97a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z" Oct 30 13:30:00.566357 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.547170 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f23416bb-8d3b-422e-9880-fca313dfd70c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889450-bnskt\" (UID: \"f23416bb-8d3b-422e-9880-fca313dfd70c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-bnskt" Oct 30 13:30:00.566357 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.564068 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ba901-29889450-fs87w"] Oct 30 13:30:00.648682 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.623363 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f88ff2-f7d4-4207-b2d1-a49a6f052a2c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg\" (UID: \"65f88ff2-f7d4-4207-b2d1-a49a6f052a2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg" Oct 30 13:30:00.658360 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.652652 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96p5j\" (UniqueName: \"kubernetes.io/projected/f23416bb-8d3b-422e-9880-fca313dfd70c-kube-api-access-96p5j\") pod \"maas-api-key-cleanup-29889450-bnskt\" (UID: \"f23416bb-8d3b-422e-9880-fca313dfd70c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-bnskt" Oct 30 13:30:00.766294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.756408 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ba901-29889450-fs87w" Oct 30 13:30:00.766294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.757961 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f23416bb-8d3b-422e-9880-fca313dfd70c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889450-bnskt\" (UID: \"f23416bb-8d3b-422e-9880-fca313dfd70c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-bnskt" Oct 30 13:30:00.766294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.758020 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbzcd\" (UniqueName: \"kubernetes.io/projected/775cb414-3b30-4adb-a746-d8a9fb548b48-kube-api-access-lbzcd\") pod \"maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj\" (UID: \"775cb414-3b30-4adb-a746-d8a9fb548b48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj" Oct 30 13:30:00.766294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.758122 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96p5j\" (UniqueName: \"kubernetes.io/projected/f23416bb-8d3b-422e-9880-fca313dfd70c-kube-api-access-96p5j\") pod \"maas-api-key-cleanup-29889450-bnskt\" (UID: \"f23416bb-8d3b-422e-9880-fca313dfd70c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-bnskt" Oct 30 13:30:00.766294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.758158 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3d54ed7-7658-49d1-95e4-2e522546e8b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-09db16-29889450-7dzx8\" (UID: \"a3d54ed7-7658-49d1-95e4-2e522546e8b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889450-7dzx8" Oct 30 13:30:00.766294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.758371 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9kbm\" (UniqueName: \"kubernetes.io/projected/a3d54ed7-7658-49d1-95e4-2e522546e8b3-kube-api-access-w9kbm\") pod \"maas-api-key-cleanup-e2e-worker-w3-09db16-29889450-7dzx8\" (UID: \"a3d54ed7-7658-49d1-95e4-2e522546e8b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889450-7dzx8" Oct 30 13:30:00.766294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.758465 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5592ca1-ccf3-4297-bc42-ea5aa0cca97a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z\" (UID: \"a5592ca1-ccf3-4297-bc42-ea5aa0cca97a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z" Oct 30 13:30:00.766294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.758503 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/775cb414-3b30-4adb-a746-d8a9fb548b48-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj\" (UID: \"775cb414-3b30-4adb-a746-d8a9fb548b48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj" Oct 30 13:30:00.766294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.758530 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6f6b\" (UniqueName: \"kubernetes.io/projected/a5592ca1-ccf3-4297-bc42-ea5aa0cca97a-kube-api-access-v6f6b\") pod \"maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z\" (UID: \"a5592ca1-ccf3-4297-bc42-ea5aa0cca97a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z" Oct 30 13:30:00.766294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.759009 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb" Oct 30 13:30:00.862938 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.799738 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f23416bb-8d3b-422e-9880-fca313dfd70c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889450-bnskt\" (UID: \"f23416bb-8d3b-422e-9880-fca313dfd70c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-bnskt" Oct 30 13:30:00.862938 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.800756 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5592ca1-ccf3-4297-bc42-ea5aa0cca97a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z\" (UID: \"a5592ca1-ccf3-4297-bc42-ea5aa0cca97a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z" Oct 30 13:30:00.862938 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.831710 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd" Oct 30 13:30:00.890289 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.888318 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9kbm\" (UniqueName: \"kubernetes.io/projected/a3d54ed7-7658-49d1-95e4-2e522546e8b3-kube-api-access-w9kbm\") pod \"maas-api-key-cleanup-e2e-worker-w3-09db16-29889450-7dzx8\" (UID: \"a3d54ed7-7658-49d1-95e4-2e522546e8b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889450-7dzx8" Oct 30 13:30:00.890289 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.888437 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/775cb414-3b30-4adb-a746-d8a9fb548b48-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj\" (UID: \"775cb414-3b30-4adb-a746-d8a9fb548b48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj" Oct 30 13:30:00.890289 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.888472 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07b99b5b-b3b4-4621-9382-4db0db4bf317-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1ba901-29889450-fs87w\" (UID: \"07b99b5b-b3b4-4621-9382-4db0db4bf317\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ba901-29889450-fs87w" Oct 30 13:30:00.890289 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.888505 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtp7g\" (UniqueName: \"kubernetes.io/projected/07b99b5b-b3b4-4621-9382-4db0db4bf317-kube-api-access-qtp7g\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1ba901-29889450-fs87w\" (UID: \"07b99b5b-b3b4-4621-9382-4db0db4bf317\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ba901-29889450-fs87w" Oct 30 13:30:00.890289 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.888572 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbzcd\" (UniqueName: \"kubernetes.io/projected/775cb414-3b30-4adb-a746-d8a9fb548b48-kube-api-access-lbzcd\") pod \"maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj\" (UID: \"775cb414-3b30-4adb-a746-d8a9fb548b48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj" Oct 30 13:30:00.890289 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.888667 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3d54ed7-7658-49d1-95e4-2e522546e8b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-09db16-29889450-7dzx8\" (UID: \"a3d54ed7-7658-49d1-95e4-2e522546e8b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889450-7dzx8" Oct 30 13:30:00.890289 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.889679 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3d54ed7-7658-49d1-95e4-2e522546e8b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-09db16-29889450-7dzx8\" (UID: \"a3d54ed7-7658-49d1-95e4-2e522546e8b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889450-7dzx8" Oct 30 13:30:00.890829 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:00.890745 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/775cb414-3b30-4adb-a746-d8a9fb548b48-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj\" (UID: \"775cb414-3b30-4adb-a746-d8a9fb548b48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj" Oct 30 13:30:01.026945 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:01.020332 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6f6b\" (UniqueName: \"kubernetes.io/projected/a5592ca1-ccf3-4297-bc42-ea5aa0cca97a-kube-api-access-v6f6b\") pod \"maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z\" (UID: \"a5592ca1-ccf3-4297-bc42-ea5aa0cca97a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z" Oct 30 13:30:01.028222 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:01.028196 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889450-bnskt"] Oct 30 13:30:01.029065 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:01.029033 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5420598-fmxqg" event={"ID":"b0716cec-3e99-4ef4-ad57-61dc8bfc9c75","Type":"ContainerStarted","Data":"97dd63bbb30d80d76ef3d3200d795cb9960769ad90db60d882a6f3773b94fe94"} Oct 30 13:30:01.029321 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:01.029098 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07b99b5b-b3b4-4621-9382-4db0db4bf317-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1ba901-29889450-fs87w\" (UID: \"07b99b5b-b3b4-4621-9382-4db0db4bf317\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ba901-29889450-fs87w" Oct 30 13:30:01.029489 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:01.029468 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtp7g\" (UniqueName: \"kubernetes.io/projected/07b99b5b-b3b4-4621-9382-4db0db4bf317-kube-api-access-qtp7g\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1ba901-29889450-fs87w\" (UID: \"07b99b5b-b3b4-4621-9382-4db0db4bf317\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ba901-29889450-fs87w" Oct 30 13:30:01.040091 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:01.040063 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9kbm\" (UniqueName: \"kubernetes.io/projected/a3d54ed7-7658-49d1-95e4-2e522546e8b3-kube-api-access-w9kbm\") pod \"maas-api-key-cleanup-e2e-worker-w3-09db16-29889450-7dzx8\" (UID: \"a3d54ed7-7658-49d1-95e4-2e522546e8b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889450-7dzx8" Oct 30 13:30:01.047082 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:01.047059 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889450-7dzx8"] Oct 30 13:30:01.048455 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:01.047763 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07b99b5b-b3b4-4621-9382-4db0db4bf317-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1ba901-29889450-fs87w\" (UID: \"07b99b5b-b3b4-4621-9382-4db0db4bf317\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ba901-29889450-fs87w" Oct 30 13:30:01.057505 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:01.057474 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z" Oct 30 13:30:01.067403 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:01.067362 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj"] Oct 30 13:30:01.076360 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:01.076320 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ba901-29889450-fs87w"] Oct 30 13:30:01.091300 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:01.091212 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96p5j\" (UniqueName: \"kubernetes.io/projected/f23416bb-8d3b-422e-9880-fca313dfd70c-kube-api-access-96p5j\") pod \"maas-api-key-cleanup-29889450-bnskt\" (UID: \"f23416bb-8d3b-422e-9880-fca313dfd70c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-bnskt" Oct 30 13:30:01.093000 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:01.092938 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxm7r\" (UniqueName: \"kubernetes.io/projected/65f88ff2-f7d4-4207-b2d1-a49a6f052a2c-kube-api-access-hxm7r\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg\" (UID: \"65f88ff2-f7d4-4207-b2d1-a49a6f052a2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg" Oct 30 13:30:01.100254 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:01.097499 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtp7g\" (UniqueName: \"kubernetes.io/projected/07b99b5b-b3b4-4621-9382-4db0db4bf317-kube-api-access-qtp7g\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1ba901-29889450-fs87w\" (UID: \"07b99b5b-b3b4-4621-9382-4db0db4bf317\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ba901-29889450-fs87w" Oct 30 13:30:01.100254 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:01.099388 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5420598-fmxqg"] Oct 30 13:30:01.117144 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:01.114157 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg" Oct 30 13:30:01.174246 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:01.174205 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbzcd\" (UniqueName: \"kubernetes.io/projected/775cb414-3b30-4adb-a746-d8a9fb548b48-kube-api-access-lbzcd\") pod \"maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj\" (UID: \"775cb414-3b30-4adb-a746-d8a9fb548b48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj" Oct 30 13:30:01.191405 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:01.190756 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889450-7dzx8" Oct 30 13:30:01.277052 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:01.276954 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ba901-29889450-fs87w" Oct 30 13:30:01.304771 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:01.304627 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq"] Oct 30 13:30:01.357979 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:01.357777 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj" Oct 30 13:30:01.358351 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:01.358225 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-bnskt" Oct 30 13:30:02.068961 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:02.068926 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq" event={"ID":"32712e88-00fd-4f97-882a-a633da6a16df","Type":"ContainerStarted","Data":"0921449db53735b721eb7ef626a48f4c8cc569a8b64257f51d994142dd226f28"} Oct 30 13:30:02.083735 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:02.083699 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5420598-fmxqg" event={"ID":"b0716cec-3e99-4ef4-ad57-61dc8bfc9c75","Type":"ContainerStarted","Data":"a2101968fc63fa24074512a15d582be91d2fb5c020ff6c32058e82da8ad07edc"} Oct 30 13:30:02.146034 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:02.145984 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5420598-fmxqg" podStartSLOduration=3.145972372 podStartE2EDuration="3.145972372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:02.144822742 +0000 UTC m=+2700.748321188" watchObservedRunningTime="2026-10-30 13:30:02.145972372 +0000 UTC m=+2700.749470817" Oct 30 13:30:02.183784 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:02.181147 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t"] Oct 30 13:30:02.184786 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:30:02.184758 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod122219fe_8078_42d9_b4ed_7c0333b9013c.slice/crio-78d1e3688c874a2c96211314709ea9482a790d094ee79340f3dba57c4b33da41 WatchSource:0}: Error finding container 78d1e3688c874a2c96211314709ea9482a790d094ee79340f3dba57c4b33da41: Status 404 returned error can't find the container with id 78d1e3688c874a2c96211314709ea9482a790d094ee79340f3dba57c4b33da41 Oct 30 13:30:02.185979 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:02.185956 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/2.log" Oct 30 13:30:02.188962 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:02.188947 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/2.log" Oct 30 13:30:02.195729 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:02.195603 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb"] Oct 30 13:30:02.208345 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:02.204723 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z"] Oct 30 13:30:02.345573 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:02.345539 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg"] Oct 30 13:30:02.386810 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:02.386781 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj"] Oct 30 13:30:02.430373 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:02.430344 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd"] Oct 30 13:30:02.539179 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:02.539147 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889450-7dzx8"] Oct 30 13:30:02.592463 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:02.592438 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ba901-29889450-fs87w"] Oct 30 13:30:02.620319 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:02.620293 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889450-bnskt"] Oct 30 13:30:02.646567 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:30:02.646359 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3d54ed7_7658_49d1_95e4_2e522546e8b3.slice/crio-7548da559110c9c0ac82482bbd37c71e8c0b895041fe891c6f615b27a937dcea WatchSource:0}: Error finding container 7548da559110c9c0ac82482bbd37c71e8c0b895041fe891c6f615b27a937dcea: Status 404 returned error can't find the container with id 7548da559110c9c0ac82482bbd37c71e8c0b895041fe891c6f615b27a937dcea Oct 30 13:30:02.647643 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:30:02.647560 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b99b5b_b3b4_4621_9382_4db0db4bf317.slice/crio-995f1732fcfbf7c8c17b50040ef60208001d80cd377d29130298e023b5d6b72b WatchSource:0}: Error finding container 995f1732fcfbf7c8c17b50040ef60208001d80cd377d29130298e023b5d6b72b: Status 404 returned error can't find the container with id 995f1732fcfbf7c8c17b50040ef60208001d80cd377d29130298e023b5d6b72b Oct 30 13:30:02.736779 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:30:02.736736 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf23416bb_8d3b_422e_9880_fca313dfd70c.slice/crio-9370dc490004a3558068a16d003bfe233d54a1f702e99f9a07fc83a738686a46 WatchSource:0}: Error finding container 9370dc490004a3558068a16d003bfe233d54a1f702e99f9a07fc83a738686a46: Status 404 returned error can't find the container with id 9370dc490004a3558068a16d003bfe233d54a1f702e99f9a07fc83a738686a46 Oct 30 13:30:03.094995 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:03.094950 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z" event={"ID":"a5592ca1-ccf3-4297-bc42-ea5aa0cca97a","Type":"ContainerStarted","Data":"91c48501dedfad70c24b924f88ff2ce76d3cf140dc54cb908b33c0036ee3f4e8"} Oct 30 13:30:03.095543 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:03.095021 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z" event={"ID":"a5592ca1-ccf3-4297-bc42-ea5aa0cca97a","Type":"ContainerStarted","Data":"7b363efa746f9af5f4eb7966fd8c4c723a0bc906beec84947408fc737eef705d"} Oct 30 13:30:03.096914 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:03.096888 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg" event={"ID":"65f88ff2-f7d4-4207-b2d1-a49a6f052a2c","Type":"ContainerStarted","Data":"97806369346a2245f9a28befe8c38d689bcc41339bf0736068f4232fc4b5017d"} Oct 30 13:30:03.097044 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:03.096936 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg" event={"ID":"65f88ff2-f7d4-4207-b2d1-a49a6f052a2c","Type":"ContainerStarted","Data":"6947ed962fa149ecd185c5a8e656e197a28e7023ab3c2feb3ea3a57dfb53aa9f"} Oct 30 13:30:03.098640 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:03.098616 4931 generic.go:356] "Generic (PLEG): container finished" podID="b0716cec-3e99-4ef4-ad57-61dc8bfc9c75" containerID="a2101968fc63fa24074512a15d582be91d2fb5c020ff6c32058e82da8ad07edc" exitCode=0 Oct 30 13:30:03.098772 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:03.098688 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5420598-fmxqg" event={"ID":"b0716cec-3e99-4ef4-ad57-61dc8bfc9c75","Type":"ContainerDied","Data":"a2101968fc63fa24074512a15d582be91d2fb5c020ff6c32058e82da8ad07edc"} Oct 30 13:30:03.100373 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:03.100342 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb" event={"ID":"52d65cd3-640f-46e7-90e9-534c3c990930","Type":"ContainerStarted","Data":"383e0d4770f3274e1b8b3865b0425a5fb0541ddc5bdcf65b67184895b3c35fe2"} Oct 30 13:30:03.100487 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:03.100391 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb" event={"ID":"52d65cd3-640f-46e7-90e9-534c3c990930","Type":"ContainerStarted","Data":"7a2770b089b697be63eb7ba3344384b639fb5032b98ae129e105827cf0e16487"} Oct 30 13:30:03.102199 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:03.102171 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj" event={"ID":"775cb414-3b30-4adb-a746-d8a9fb548b48","Type":"ContainerStarted","Data":"093a4474504355abc355b76b46867faf8161fe945c5af6e428058031098fbcd5"} Oct 30 13:30:03.102334 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:03.102223 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj" event={"ID":"775cb414-3b30-4adb-a746-d8a9fb548b48","Type":"ContainerStarted","Data":"685a4ddb43edbe2ae6cc5710d59eb693f9b3829e3588576b0a442e2728b5b06f"} Oct 30 13:30:03.103689 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:03.103667 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-bnskt" event={"ID":"f23416bb-8d3b-422e-9880-fca313dfd70c","Type":"ContainerStarted","Data":"9370dc490004a3558068a16d003bfe233d54a1f702e99f9a07fc83a738686a46"} Oct 30 13:30:03.105354 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:03.105333 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq" event={"ID":"32712e88-00fd-4f97-882a-a633da6a16df","Type":"ContainerStarted","Data":"fe6342f1239be6df9315215e5d69d8e44851aef163c00cc329cc46317ed331e1"} Oct 30 13:30:03.106670 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:03.106645 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889450-7dzx8" event={"ID":"a3d54ed7-7658-49d1-95e4-2e522546e8b3","Type":"ContainerStarted","Data":"7548da559110c9c0ac82482bbd37c71e8c0b895041fe891c6f615b27a937dcea"} Oct 30 13:30:03.108238 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:03.108217 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t" event={"ID":"122219fe-8078-42d9-b4ed-7c0333b9013c","Type":"ContainerStarted","Data":"86b308f0e6d2d815a895a0a84dd99a406500998306b7e2c09b29b7c57a85033f"} Oct 30 13:30:03.108371 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:03.108251 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t" event={"ID":"122219fe-8078-42d9-b4ed-7c0333b9013c","Type":"ContainerStarted","Data":"78d1e3688c874a2c96211314709ea9482a790d094ee79340f3dba57c4b33da41"} Oct 30 13:30:03.109576 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:03.109551 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ba901-29889450-fs87w" event={"ID":"07b99b5b-b3b4-4621-9382-4db0db4bf317","Type":"ContainerStarted","Data":"995f1732fcfbf7c8c17b50040ef60208001d80cd377d29130298e023b5d6b72b"} Oct 30 13:30:03.111348 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:03.111321 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd" event={"ID":"ae4551d9-3f2a-4278-b567-243121f8e1ff","Type":"ContainerStarted","Data":"a1212869d971d9daea18586c74c7d39a2e6404d1b22dd82dfe07ae323e8efcae"} Oct 30 13:30:03.111423 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:03.111375 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd" event={"ID":"ae4551d9-3f2a-4278-b567-243121f8e1ff","Type":"ContainerStarted","Data":"61cdb499cfaf640b09a58e3df71c061477551182189b5d889da9162a9a885910"} Oct 30 13:30:03.146130 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:03.146088 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z" podStartSLOduration=3.1460751829999998 podStartE2EDuration="3.146075183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:03.137854058 +0000 UTC m=+2701.741352505" watchObservedRunningTime="2026-10-30 13:30:03.146075183 +0000 UTC m=+2701.749573629" Oct 30 13:30:03.181139 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:03.181093 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t" podStartSLOduration=3.181076207 podStartE2EDuration="3.181076207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:03.179469823 +0000 UTC m=+2701.782968269" watchObservedRunningTime="2026-10-30 13:30:03.181076207 +0000 UTC m=+2701.784574653" Oct 30 13:30:03.228507 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:03.228406 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq" podStartSLOduration=3.228391756 podStartE2EDuration="3.228391756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:03.223715321 +0000 UTC m=+2701.827213789" watchObservedRunningTime="2026-10-30 13:30:03.228391756 +0000 UTC m=+2701.831890234" Oct 30 13:30:03.270503 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:03.270446 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb" podStartSLOduration=3.270432878 podStartE2EDuration="3.270432878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:03.267737296 +0000 UTC m=+2701.871235745" watchObservedRunningTime="2026-10-30 13:30:03.270432878 +0000 UTC m=+2701.873931323" Oct 30 13:30:03.313220 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:03.313164 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd" podStartSLOduration=3.313146486 podStartE2EDuration="3.313146486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:03.31054223 +0000 UTC m=+2701.914040675" watchObservedRunningTime="2026-10-30 13:30:03.313146486 +0000 UTC m=+2701.916644932" Oct 30 13:30:03.424690 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:03.424634 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg" podStartSLOduration=3.4246218920000002 podStartE2EDuration="3.424621892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:03.418649715 +0000 UTC m=+2702.022148162" watchObservedRunningTime="2026-10-30 13:30:03.424621892 +0000 UTC m=+2702.028120339" Oct 30 13:30:03.476330 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:03.476260 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj" podStartSLOduration=3.4762442829999998 podStartE2EDuration="3.476244283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:03.47107876 +0000 UTC m=+2702.074577207" watchObservedRunningTime="2026-10-30 13:30:03.476244283 +0000 UTC m=+2702.079742728" Oct 30 13:30:04.123885 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.123851 4931 generic.go:356] "Generic (PLEG): container finished" podID="52d65cd3-640f-46e7-90e9-534c3c990930" containerID="383e0d4770f3274e1b8b3865b0425a5fb0541ddc5bdcf65b67184895b3c35fe2" exitCode=60 Oct 30 13:30:04.124568 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.124492 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb" event={"ID":"52d65cd3-640f-46e7-90e9-534c3c990930","Type":"ContainerDied","Data":"383e0d4770f3274e1b8b3865b0425a5fb0541ddc5bdcf65b67184895b3c35fe2"} Oct 30 13:30:04.124681 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.124663 4931 scope.go:122] "RemoveContainer" containerID="383e0d4770f3274e1b8b3865b0425a5fb0541ddc5bdcf65b67184895b3c35fe2" Oct 30 13:30:04.127196 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.127170 4931 generic.go:356] "Generic (PLEG): container finished" podID="775cb414-3b30-4adb-a746-d8a9fb548b48" containerID="093a4474504355abc355b76b46867faf8161fe945c5af6e428058031098fbcd5" exitCode=60 Oct 30 13:30:04.127310 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.127251 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj" event={"ID":"775cb414-3b30-4adb-a746-d8a9fb548b48","Type":"ContainerDied","Data":"093a4474504355abc355b76b46867faf8161fe945c5af6e428058031098fbcd5"} Oct 30 13:30:04.127595 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.127579 4931 scope.go:122] "RemoveContainer" containerID="093a4474504355abc355b76b46867faf8161fe945c5af6e428058031098fbcd5" Oct 30 13:30:04.130581 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.130561 4931 generic.go:356] "Generic (PLEG): container finished" podID="f23416bb-8d3b-422e-9880-fca313dfd70c" containerID="6c057e215f1f1d7226f8ed1dd7cd445b27162070ec03541d7abc51a98dfd3da9" exitCode=60 Oct 30 13:30:04.130707 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.130686 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-bnskt" event={"ID":"f23416bb-8d3b-422e-9880-fca313dfd70c","Type":"ContainerDied","Data":"6c057e215f1f1d7226f8ed1dd7cd445b27162070ec03541d7abc51a98dfd3da9"} Oct 30 13:30:04.130958 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.130918 4931 scope.go:122] "RemoveContainer" containerID="6c057e215f1f1d7226f8ed1dd7cd445b27162070ec03541d7abc51a98dfd3da9" Oct 30 13:30:04.133002 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.132978 4931 generic.go:356] "Generic (PLEG): container finished" podID="32712e88-00fd-4f97-882a-a633da6a16df" containerID="fe6342f1239be6df9315215e5d69d8e44851aef163c00cc329cc46317ed331e1" exitCode=60 Oct 30 13:30:04.133100 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.133062 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq" event={"ID":"32712e88-00fd-4f97-882a-a633da6a16df","Type":"ContainerDied","Data":"fe6342f1239be6df9315215e5d69d8e44851aef163c00cc329cc46317ed331e1"} Oct 30 13:30:04.133358 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.133339 4931 scope.go:122] "RemoveContainer" containerID="fe6342f1239be6df9315215e5d69d8e44851aef163c00cc329cc46317ed331e1" Oct 30 13:30:04.135118 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.135092 4931 generic.go:356] "Generic (PLEG): container finished" podID="a3d54ed7-7658-49d1-95e4-2e522546e8b3" containerID="d5dcfd31c60e3c3fbca2a68eaa78cf563a9e00f9bd7ea41aed481c1f47450e3f" exitCode=60 Oct 30 13:30:04.135233 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.135177 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889450-7dzx8" event={"ID":"a3d54ed7-7658-49d1-95e4-2e522546e8b3","Type":"ContainerDied","Data":"d5dcfd31c60e3c3fbca2a68eaa78cf563a9e00f9bd7ea41aed481c1f47450e3f"} Oct 30 13:30:04.135554 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.135535 4931 scope.go:122] "RemoveContainer" containerID="d5dcfd31c60e3c3fbca2a68eaa78cf563a9e00f9bd7ea41aed481c1f47450e3f" Oct 30 13:30:04.137445 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.137427 4931 generic.go:356] "Generic (PLEG): container finished" podID="122219fe-8078-42d9-b4ed-7c0333b9013c" containerID="86b308f0e6d2d815a895a0a84dd99a406500998306b7e2c09b29b7c57a85033f" exitCode=60 Oct 30 13:30:04.137547 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.137503 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t" event={"ID":"122219fe-8078-42d9-b4ed-7c0333b9013c","Type":"ContainerDied","Data":"86b308f0e6d2d815a895a0a84dd99a406500998306b7e2c09b29b7c57a85033f"} Oct 30 13:30:04.137696 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.137678 4931 scope.go:122] "RemoveContainer" containerID="86b308f0e6d2d815a895a0a84dd99a406500998306b7e2c09b29b7c57a85033f" Oct 30 13:30:04.139376 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.139357 4931 generic.go:356] "Generic (PLEG): container finished" podID="07b99b5b-b3b4-4621-9382-4db0db4bf317" containerID="4a3b5cc9f898d83c17bb79dfe4a3e2f726ca578e24c0450662b8c1600347e7b6" exitCode=60 Oct 30 13:30:04.139464 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.139381 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ba901-29889450-fs87w" event={"ID":"07b99b5b-b3b4-4621-9382-4db0db4bf317","Type":"ContainerDied","Data":"4a3b5cc9f898d83c17bb79dfe4a3e2f726ca578e24c0450662b8c1600347e7b6"} Oct 30 13:30:04.139662 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.139646 4931 scope.go:122] "RemoveContainer" containerID="4a3b5cc9f898d83c17bb79dfe4a3e2f726ca578e24c0450662b8c1600347e7b6" Oct 30 13:30:04.152808 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.152782 4931 generic.go:356] "Generic (PLEG): container finished" podID="ae4551d9-3f2a-4278-b567-243121f8e1ff" containerID="a1212869d971d9daea18586c74c7d39a2e6404d1b22dd82dfe07ae323e8efcae" exitCode=60 Oct 30 13:30:04.152987 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.152920 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd" event={"ID":"ae4551d9-3f2a-4278-b567-243121f8e1ff","Type":"ContainerDied","Data":"a1212869d971d9daea18586c74c7d39a2e6404d1b22dd82dfe07ae323e8efcae"} Oct 30 13:30:04.154085 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.153301 4931 scope.go:122] "RemoveContainer" containerID="a1212869d971d9daea18586c74c7d39a2e6404d1b22dd82dfe07ae323e8efcae" Oct 30 13:30:04.162410 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.162383 4931 generic.go:356] "Generic (PLEG): container finished" podID="a5592ca1-ccf3-4297-bc42-ea5aa0cca97a" containerID="91c48501dedfad70c24b924f88ff2ce76d3cf140dc54cb908b33c0036ee3f4e8" exitCode=60 Oct 30 13:30:04.162514 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.162466 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z" event={"ID":"a5592ca1-ccf3-4297-bc42-ea5aa0cca97a","Type":"ContainerDied","Data":"91c48501dedfad70c24b924f88ff2ce76d3cf140dc54cb908b33c0036ee3f4e8"} Oct 30 13:30:04.162838 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.162813 4931 scope.go:122] "RemoveContainer" containerID="91c48501dedfad70c24b924f88ff2ce76d3cf140dc54cb908b33c0036ee3f4e8" Oct 30 13:30:04.171249 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.170668 4931 generic.go:356] "Generic (PLEG): container finished" podID="65f88ff2-f7d4-4207-b2d1-a49a6f052a2c" containerID="97806369346a2245f9a28befe8c38d689bcc41339bf0736068f4232fc4b5017d" exitCode=60 Oct 30 13:30:04.171884 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.171664 4931 scope.go:122] "RemoveContainer" containerID="97806369346a2245f9a28befe8c38d689bcc41339bf0736068f4232fc4b5017d" Oct 30 13:30:04.172030 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.172003 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg" event={"ID":"65f88ff2-f7d4-4207-b2d1-a49a6f052a2c","Type":"ContainerDied","Data":"97806369346a2245f9a28befe8c38d689bcc41339bf0736068f4232fc4b5017d"} Oct 30 13:30:04.655045 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.655020 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5420598-fmxqg" Oct 30 13:30:04.720665 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.720633 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0716cec-3e99-4ef4-ad57-61dc8bfc9c75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0716cec-3e99-4ef4-ad57-61dc8bfc9c75-maas-api-xks-ca\") pod \"b0716cec-3e99-4ef4-ad57-61dc8bfc9c75\" (UID: \"b0716cec-3e99-4ef4-ad57-61dc8bfc9c75\") " Oct 30 13:30:04.720832 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.720672 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0716cec-3e99-4ef4-ad57-61dc8bfc9c75-kube-api-access-jk4gp\" (UniqueName: \"kubernetes.io/projected/b0716cec-3e99-4ef4-ad57-61dc8bfc9c75-kube-api-access-jk4gp\") pod \"b0716cec-3e99-4ef4-ad57-61dc8bfc9c75\" (UID: \"b0716cec-3e99-4ef4-ad57-61dc8bfc9c75\") " Oct 30 13:30:04.720832 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.720784 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0716cec-3e99-4ef4-ad57-61dc8bfc9c75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0716cec-3e99-4ef4-ad57-61dc8bfc9c75-maas-api-openshift-ca\") pod \"b0716cec-3e99-4ef4-ad57-61dc8bfc9c75\" (UID: \"b0716cec-3e99-4ef4-ad57-61dc8bfc9c75\") " Oct 30 13:30:04.721171 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.721147 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0716cec-3e99-4ef4-ad57-61dc8bfc9c75-maas-api-openshift-ca" pod "b0716cec-3e99-4ef4-ad57-61dc8bfc9c75" (UID: "b0716cec-3e99-4ef4-ad57-61dc8bfc9c75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:04.721296 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.721242 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0716cec-3e99-4ef4-ad57-61dc8bfc9c75-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:30:04.723029 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.723005 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0716cec-3e99-4ef4-ad57-61dc8bfc9c75-maas-api-xks-ca" pod "b0716cec-3e99-4ef4-ad57-61dc8bfc9c75" (UID: "b0716cec-3e99-4ef4-ad57-61dc8bfc9c75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:30:04.723116 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.723055 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0716cec-3e99-4ef4-ad57-61dc8bfc9c75-kube-api-access-jk4gp" pod "b0716cec-3e99-4ef4-ad57-61dc8bfc9c75" (UID: "b0716cec-3e99-4ef4-ad57-61dc8bfc9c75"). InnerVolumeSpecName "kube-api-access-jk4gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:04.822883 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.822843 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0716cec-3e99-4ef4-ad57-61dc8bfc9c75-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:30:04.822883 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:04.822884 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk4gp\" (UniqueName: \"kubernetes.io/projected/b0716cec-3e99-4ef4-ad57-61dc8bfc9c75-kube-api-access-jk4gp\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:30:05.181511 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:05.181466 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t" event={"ID":"122219fe-8078-42d9-b4ed-7c0333b9013c","Type":"ContainerStarted","Data":"584c9a6962fa63ac7b274e0ffd328569257286cbe4c42289e7822daca55b9096"} Oct 30 13:30:05.184191 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:05.184163 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ba901-29889450-fs87w" event={"ID":"07b99b5b-b3b4-4621-9382-4db0db4bf317","Type":"ContainerStarted","Data":"7dfea41cb9bf6049e289ac1826948e9ae06edadf9bd296468138a466904a6bb9"} Oct 30 13:30:05.186091 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:05.186066 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd" event={"ID":"ae4551d9-3f2a-4278-b567-243121f8e1ff","Type":"ContainerStarted","Data":"bbe0d45e37d5b0a91811f7561b098ac131672db1eeb7f8ac49202fdb44fab745"} Oct 30 13:30:05.188105 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:05.188076 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z" event={"ID":"a5592ca1-ccf3-4297-bc42-ea5aa0cca97a","Type":"ContainerStarted","Data":"8002b5f56b7061f96ec1e7ae7f3d11f43ba5bc56b367f09052036a368a1e204d"} Oct 30 13:30:05.190539 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:05.190482 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg" event={"ID":"65f88ff2-f7d4-4207-b2d1-a49a6f052a2c","Type":"ContainerStarted","Data":"fe52f278f1292402dd28b5be8de766a2b3724296eefee34b317a221867bd3d41"} Oct 30 13:30:05.191878 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:05.191854 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5420598-fmxqg" Oct 30 13:30:05.191979 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:05.191872 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5420598-fmxqg" event={"ID":"b0716cec-3e99-4ef4-ad57-61dc8bfc9c75","Type":"ContainerDied","Data":"97dd63bbb30d80d76ef3d3200d795cb9960769ad90db60d882a6f3773b94fe94"} Oct 30 13:30:05.191979 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:05.191910 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97dd63bbb30d80d76ef3d3200d795cb9960769ad90db60d882a6f3773b94fe94" Oct 30 13:30:05.193892 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:05.193861 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb" event={"ID":"52d65cd3-640f-46e7-90e9-534c3c990930","Type":"ContainerStarted","Data":"2ec30021a01f3e7b0f3cee253bbc3ec41ff78d8daae2562a3e8fd6eb0924f7d5"} Oct 30 13:30:05.195883 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:05.195859 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj" event={"ID":"775cb414-3b30-4adb-a746-d8a9fb548b48","Type":"ContainerStarted","Data":"ebb797d61af0ecfd0faf4e35994322dbce96f9b7b044ab736aba22315af423c6"} Oct 30 13:30:05.197762 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:05.197734 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-bnskt" event={"ID":"f23416bb-8d3b-422e-9880-fca313dfd70c","Type":"ContainerStarted","Data":"8bd1bc09525560afc1fac5ef85feba7a27d5be5822f13753d3a44edc46245def"} Oct 30 13:30:05.199443 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:05.199424 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq" event={"ID":"32712e88-00fd-4f97-882a-a633da6a16df","Type":"ContainerStarted","Data":"11f0e27327c96aad37aa513810967da717e117b60d2c8fb63c4b5d1194fd5b4f"} Oct 30 13:30:05.201909 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:05.201862 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889450-7dzx8" event={"ID":"a3d54ed7-7658-49d1-95e4-2e522546e8b3","Type":"ContainerStarted","Data":"3a9f43f02d38aab7831805fcf5ed75451d8aeada6df9b18681b236aa6242787f"} Oct 30 13:30:05.615968 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:05.615904 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889450-7dzx8" podStartSLOduration=5.615885906 podStartE2EDuration="5.615885906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:05.570245951 +0000 UTC m=+2704.173744399" watchObservedRunningTime="2026-10-30 13:30:05.615885906 +0000 UTC m=+2704.219384353" Oct 30 13:30:05.661387 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:05.661334 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-bnskt" podStartSLOduration=5.661318102 podStartE2EDuration="5.661318102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:05.661216381 +0000 UTC m=+2704.264714828" watchObservedRunningTime="2026-10-30 13:30:05.661318102 +0000 UTC m=+2704.264816550" Oct 30 13:30:05.709681 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:05.709600 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ba901-29889450-fs87w" podStartSLOduration=5.7095860080000005 podStartE2EDuration="5.709586008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:05.706220734 +0000 UTC m=+2704.309719180" watchObservedRunningTime="2026-10-30 13:30:05.709586008 +0000 UTC m=+2704.313084454" Oct 30 13:30:06.218129 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.218091 4931 generic.go:356] "Generic (PLEG): container finished" podID="52d65cd3-640f-46e7-90e9-534c3c990930" containerID="2ec30021a01f3e7b0f3cee253bbc3ec41ff78d8daae2562a3e8fd6eb0924f7d5" exitCode=60 Oct 30 13:30:06.218651 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.218231 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb" event={"ID":"52d65cd3-640f-46e7-90e9-534c3c990930","Type":"ContainerDied","Data":"2ec30021a01f3e7b0f3cee253bbc3ec41ff78d8daae2562a3e8fd6eb0924f7d5"} Oct 30 13:30:06.218651 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.218341 4931 scope.go:122] "RemoveContainer" containerID="383e0d4770f3274e1b8b3865b0425a5fb0541ddc5bdcf65b67184895b3c35fe2" Oct 30 13:30:06.218651 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.218540 4931 scope.go:122] "RemoveContainer" containerID="2ec30021a01f3e7b0f3cee253bbc3ec41ff78d8daae2562a3e8fd6eb0924f7d5" Oct 30 13:30:06.218918 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:30:06.218894 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb_odh-ai-gateway-infra(52d65cd3-640f-46e7-90e9-534c3c990930)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb" podUID="52d65cd3-640f-46e7-90e9-534c3c990930" Oct 30 13:30:06.222208 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.222188 4931 generic.go:356] "Generic (PLEG): container finished" podID="775cb414-3b30-4adb-a746-d8a9fb548b48" containerID="ebb797d61af0ecfd0faf4e35994322dbce96f9b7b044ab736aba22315af423c6" exitCode=60 Oct 30 13:30:06.222365 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.222331 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj" event={"ID":"775cb414-3b30-4adb-a746-d8a9fb548b48","Type":"ContainerDied","Data":"ebb797d61af0ecfd0faf4e35994322dbce96f9b7b044ab736aba22315af423c6"} Oct 30 13:30:06.223409 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.223388 4931 scope.go:122] "RemoveContainer" containerID="ebb797d61af0ecfd0faf4e35994322dbce96f9b7b044ab736aba22315af423c6" Oct 30 13:30:06.223941 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:30:06.223918 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj_odh-ai-gateway-infra(775cb414-3b30-4adb-a746-d8a9fb548b48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj" podUID="775cb414-3b30-4adb-a746-d8a9fb548b48" Oct 30 13:30:06.226849 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.226824 4931 generic.go:356] "Generic (PLEG): container finished" podID="f23416bb-8d3b-422e-9880-fca313dfd70c" containerID="8bd1bc09525560afc1fac5ef85feba7a27d5be5822f13753d3a44edc46245def" exitCode=60 Oct 30 13:30:06.227089 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.226997 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-bnskt" event={"ID":"f23416bb-8d3b-422e-9880-fca313dfd70c","Type":"ContainerDied","Data":"8bd1bc09525560afc1fac5ef85feba7a27d5be5822f13753d3a44edc46245def"} Oct 30 13:30:06.229386 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.227164 4931 scope.go:122] "RemoveContainer" containerID="8bd1bc09525560afc1fac5ef85feba7a27d5be5822f13753d3a44edc46245def" Oct 30 13:30:06.229755 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:30:06.229732 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889450-bnskt_odh-ai-gateway-infra(f23416bb-8d3b-422e-9880-fca313dfd70c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-bnskt" podUID="f23416bb-8d3b-422e-9880-fca313dfd70c" Oct 30 13:30:06.232220 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.232198 4931 generic.go:356] "Generic (PLEG): container finished" podID="32712e88-00fd-4f97-882a-a633da6a16df" containerID="11f0e27327c96aad37aa513810967da717e117b60d2c8fb63c4b5d1194fd5b4f" exitCode=60 Oct 30 13:30:06.232366 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.232310 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq" event={"ID":"32712e88-00fd-4f97-882a-a633da6a16df","Type":"ContainerDied","Data":"11f0e27327c96aad37aa513810967da717e117b60d2c8fb63c4b5d1194fd5b4f"} Oct 30 13:30:06.232634 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.232603 4931 scope.go:122] "RemoveContainer" containerID="11f0e27327c96aad37aa513810967da717e117b60d2c8fb63c4b5d1194fd5b4f" Oct 30 13:30:06.233128 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:30:06.233097 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq_odh-ai-gateway-infra(32712e88-00fd-4f97-882a-a633da6a16df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq" podUID="32712e88-00fd-4f97-882a-a633da6a16df" Oct 30 13:30:06.234884 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.234860 4931 generic.go:356] "Generic (PLEG): container finished" podID="a3d54ed7-7658-49d1-95e4-2e522546e8b3" containerID="3a9f43f02d38aab7831805fcf5ed75451d8aeada6df9b18681b236aa6242787f" exitCode=60 Oct 30 13:30:06.234983 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.234954 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889450-7dzx8" event={"ID":"a3d54ed7-7658-49d1-95e4-2e522546e8b3","Type":"ContainerDied","Data":"3a9f43f02d38aab7831805fcf5ed75451d8aeada6df9b18681b236aa6242787f"} Oct 30 13:30:06.235257 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.235234 4931 scope.go:122] "RemoveContainer" containerID="3a9f43f02d38aab7831805fcf5ed75451d8aeada6df9b18681b236aa6242787f" Oct 30 13:30:06.237080 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:30:06.235849 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-09db16-29889450-7dzx8_odh-ai-gateway-infra(a3d54ed7-7658-49d1-95e4-2e522546e8b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889450-7dzx8" podUID="a3d54ed7-7658-49d1-95e4-2e522546e8b3" Oct 30 13:30:06.251492 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.242904 4931 scope.go:122] "RemoveContainer" containerID="093a4474504355abc355b76b46867faf8161fe945c5af6e428058031098fbcd5" Oct 30 13:30:06.276251 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.263935 4931 generic.go:356] "Generic (PLEG): container finished" podID="122219fe-8078-42d9-b4ed-7c0333b9013c" containerID="584c9a6962fa63ac7b274e0ffd328569257286cbe4c42289e7822daca55b9096" exitCode=60 Oct 30 13:30:06.276251 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.264081 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t" event={"ID":"122219fe-8078-42d9-b4ed-7c0333b9013c","Type":"ContainerDied","Data":"584c9a6962fa63ac7b274e0ffd328569257286cbe4c42289e7822daca55b9096"} Oct 30 13:30:06.276251 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.264665 4931 scope.go:122] "RemoveContainer" containerID="584c9a6962fa63ac7b274e0ffd328569257286cbe4c42289e7822daca55b9096" Oct 30 13:30:06.285301 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:30:06.277918 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t_odh-ai-gateway-infra(122219fe-8078-42d9-b4ed-7c0333b9013c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t" podUID="122219fe-8078-42d9-b4ed-7c0333b9013c" Oct 30 13:30:06.388553 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.388514 4931 generic.go:356] "Generic (PLEG): container finished" podID="07b99b5b-b3b4-4621-9382-4db0db4bf317" containerID="7dfea41cb9bf6049e289ac1826948e9ae06edadf9bd296468138a466904a6bb9" exitCode=60 Oct 30 13:30:06.388893 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.388871 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ba901-29889450-fs87w" event={"ID":"07b99b5b-b3b4-4621-9382-4db0db4bf317","Type":"ContainerDied","Data":"7dfea41cb9bf6049e289ac1826948e9ae06edadf9bd296468138a466904a6bb9"} Oct 30 13:30:06.389481 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.389458 4931 scope.go:122] "RemoveContainer" containerID="7dfea41cb9bf6049e289ac1826948e9ae06edadf9bd296468138a466904a6bb9" Oct 30 13:30:06.390067 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:30:06.390038 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-1ba901-29889450-fs87w_odh-ai-gateway-infra(07b99b5b-b3b4-4621-9382-4db0db4bf317)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ba901-29889450-fs87w" podUID="07b99b5b-b3b4-4621-9382-4db0db4bf317" Oct 30 13:30:06.424734 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.424698 4931 generic.go:356] "Generic (PLEG): container finished" podID="ae4551d9-3f2a-4278-b567-243121f8e1ff" containerID="bbe0d45e37d5b0a91811f7561b098ac131672db1eeb7f8ac49202fdb44fab745" exitCode=60 Oct 30 13:30:06.433238 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.428695 4931 generic.go:356] "Generic (PLEG): container finished" podID="a5592ca1-ccf3-4297-bc42-ea5aa0cca97a" containerID="8002b5f56b7061f96ec1e7ae7f3d11f43ba5bc56b367f09052036a368a1e204d" exitCode=60 Oct 30 13:30:06.433238 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.432551 4931 generic.go:356] "Generic (PLEG): container finished" podID="65f88ff2-f7d4-4207-b2d1-a49a6f052a2c" containerID="fe52f278f1292402dd28b5be8de766a2b3724296eefee34b317a221867bd3d41" exitCode=60 Oct 30 13:30:06.439786 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.433726 4931 scope.go:122] "RemoveContainer" containerID="fe52f278f1292402dd28b5be8de766a2b3724296eefee34b317a221867bd3d41" Oct 30 13:30:06.439786 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.433801 4931 scope.go:122] "RemoveContainer" containerID="bbe0d45e37d5b0a91811f7561b098ac131672db1eeb7f8ac49202fdb44fab745" Oct 30 13:30:06.439786 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:30:06.434159 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg_odh-ai-gateway-infra(65f88ff2-f7d4-4207-b2d1-a49a6f052a2c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg" podUID="65f88ff2-f7d4-4207-b2d1-a49a6f052a2c" Oct 30 13:30:06.439786 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.435039 4931 scope.go:122] "RemoveContainer" containerID="8002b5f56b7061f96ec1e7ae7f3d11f43ba5bc56b367f09052036a368a1e204d" Oct 30 13:30:06.439786 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:30:06.435449 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z_odh-ai-gateway-infra(a5592ca1-ccf3-4297-bc42-ea5aa0cca97a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z" podUID="a5592ca1-ccf3-4297-bc42-ea5aa0cca97a" Oct 30 13:30:06.444350 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.425157 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd" event={"ID":"ae4551d9-3f2a-4278-b567-243121f8e1ff","Type":"ContainerDied","Data":"bbe0d45e37d5b0a91811f7561b098ac131672db1eeb7f8ac49202fdb44fab745"} Oct 30 13:30:06.444350 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.440730 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z" event={"ID":"a5592ca1-ccf3-4297-bc42-ea5aa0cca97a","Type":"ContainerDied","Data":"8002b5f56b7061f96ec1e7ae7f3d11f43ba5bc56b367f09052036a368a1e204d"} Oct 30 13:30:06.444350 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.440764 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg" event={"ID":"65f88ff2-f7d4-4207-b2d1-a49a6f052a2c","Type":"ContainerDied","Data":"fe52f278f1292402dd28b5be8de766a2b3724296eefee34b317a221867bd3d41"} Oct 30 13:30:06.455017 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:30:06.454900 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd_odh-ai-gateway-infra(ae4551d9-3f2a-4278-b567-243121f8e1ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd" podUID="ae4551d9-3f2a-4278-b567-243121f8e1ff" Oct 30 13:30:06.455425 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.455399 4931 scope.go:122] "RemoveContainer" containerID="6c057e215f1f1d7226f8ed1dd7cd445b27162070ec03541d7abc51a98dfd3da9" Oct 30 13:30:06.493491 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.493464 4931 scope.go:122] "RemoveContainer" containerID="fe6342f1239be6df9315215e5d69d8e44851aef163c00cc329cc46317ed331e1" Oct 30 13:30:06.509141 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.509119 4931 scope.go:122] "RemoveContainer" containerID="d5dcfd31c60e3c3fbca2a68eaa78cf563a9e00f9bd7ea41aed481c1f47450e3f" Oct 30 13:30:06.524434 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.524407 4931 scope.go:122] "RemoveContainer" containerID="86b308f0e6d2d815a895a0a84dd99a406500998306b7e2c09b29b7c57a85033f" Oct 30 13:30:06.542418 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.542392 4931 scope.go:122] "RemoveContainer" containerID="4a3b5cc9f898d83c17bb79dfe4a3e2f726ca578e24c0450662b8c1600347e7b6" Oct 30 13:30:06.573256 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.573231 4931 scope.go:122] "RemoveContainer" containerID="a1212869d971d9daea18586c74c7d39a2e6404d1b22dd82dfe07ae323e8efcae" Oct 30 13:30:06.591189 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.591164 4931 scope.go:122] "RemoveContainer" containerID="91c48501dedfad70c24b924f88ff2ce76d3cf140dc54cb908b33c0036ee3f4e8" Oct 30 13:30:06.608309 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:06.608286 4931 scope.go:122] "RemoveContainer" containerID="97806369346a2245f9a28befe8c38d689bcc41339bf0736068f4232fc4b5017d" Oct 30 13:30:07.368477 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:30:07.368435 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:30:07.442556 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:07.442525 4931 scope.go:122] "RemoveContainer" containerID="2ec30021a01f3e7b0f3cee253bbc3ec41ff78d8daae2562a3e8fd6eb0924f7d5" Oct 30 13:30:07.442950 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:30:07.442918 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb_odh-ai-gateway-infra(52d65cd3-640f-46e7-90e9-534c3c990930)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb" podUID="52d65cd3-640f-46e7-90e9-534c3c990930" Oct 30 13:30:07.444439 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:07.444417 4931 scope.go:122] "RemoveContainer" containerID="ebb797d61af0ecfd0faf4e35994322dbce96f9b7b044ab736aba22315af423c6" Oct 30 13:30:07.444803 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:30:07.444778 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj_odh-ai-gateway-infra(775cb414-3b30-4adb-a746-d8a9fb548b48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj" podUID="775cb414-3b30-4adb-a746-d8a9fb548b48" Oct 30 13:30:07.446223 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:07.446201 4931 scope.go:122] "RemoveContainer" containerID="8bd1bc09525560afc1fac5ef85feba7a27d5be5822f13753d3a44edc46245def" Oct 30 13:30:07.446539 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:30:07.446520 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889450-bnskt_odh-ai-gateway-infra(f23416bb-8d3b-422e-9880-fca313dfd70c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-bnskt" podUID="f23416bb-8d3b-422e-9880-fca313dfd70c" Oct 30 13:30:07.447765 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:07.447747 4931 scope.go:122] "RemoveContainer" containerID="11f0e27327c96aad37aa513810967da717e117b60d2c8fb63c4b5d1194fd5b4f" Oct 30 13:30:07.448018 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:30:07.447987 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq_odh-ai-gateway-infra(32712e88-00fd-4f97-882a-a633da6a16df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq" podUID="32712e88-00fd-4f97-882a-a633da6a16df" Oct 30 13:30:07.449551 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:07.449529 4931 scope.go:122] "RemoveContainer" containerID="3a9f43f02d38aab7831805fcf5ed75451d8aeada6df9b18681b236aa6242787f" Oct 30 13:30:07.449815 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:30:07.449787 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-09db16-29889450-7dzx8_odh-ai-gateway-infra(a3d54ed7-7658-49d1-95e4-2e522546e8b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889450-7dzx8" podUID="a3d54ed7-7658-49d1-95e4-2e522546e8b3" Oct 30 13:30:07.451345 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:07.451327 4931 scope.go:122] "RemoveContainer" containerID="584c9a6962fa63ac7b274e0ffd328569257286cbe4c42289e7822daca55b9096" Oct 30 13:30:07.451592 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:30:07.451570 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t_odh-ai-gateway-infra(122219fe-8078-42d9-b4ed-7c0333b9013c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t" podUID="122219fe-8078-42d9-b4ed-7c0333b9013c" Oct 30 13:30:07.453173 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:07.453153 4931 scope.go:122] "RemoveContainer" containerID="7dfea41cb9bf6049e289ac1826948e9ae06edadf9bd296468138a466904a6bb9" Oct 30 13:30:07.453399 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:30:07.453377 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-1ba901-29889450-fs87w_odh-ai-gateway-infra(07b99b5b-b3b4-4621-9382-4db0db4bf317)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ba901-29889450-fs87w" podUID="07b99b5b-b3b4-4621-9382-4db0db4bf317" Oct 30 13:30:07.454929 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:07.454910 4931 scope.go:122] "RemoveContainer" containerID="bbe0d45e37d5b0a91811f7561b098ac131672db1eeb7f8ac49202fdb44fab745" Oct 30 13:30:07.455166 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:30:07.455133 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd_odh-ai-gateway-infra(ae4551d9-3f2a-4278-b567-243121f8e1ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd" podUID="ae4551d9-3f2a-4278-b567-243121f8e1ff" Oct 30 13:30:07.456979 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:07.456950 4931 scope.go:122] "RemoveContainer" containerID="8002b5f56b7061f96ec1e7ae7f3d11f43ba5bc56b367f09052036a368a1e204d" Oct 30 13:30:07.457635 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:30:07.457169 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z_odh-ai-gateway-infra(a5592ca1-ccf3-4297-bc42-ea5aa0cca97a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z" podUID="a5592ca1-ccf3-4297-bc42-ea5aa0cca97a" Oct 30 13:30:07.459387 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:07.459367 4931 scope.go:122] "RemoveContainer" containerID="fe52f278f1292402dd28b5be8de766a2b3724296eefee34b317a221867bd3d41" Oct 30 13:30:07.459629 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:30:07.459608 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg_odh-ai-gateway-infra(65f88ff2-f7d4-4207-b2d1-a49a6f052a2c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg" podUID="65f88ff2-f7d4-4207-b2d1-a49a6f052a2c" Oct 30 13:30:10.272552 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.272515 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl"] Oct 30 13:30:10.273597 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.273568 4931 cpu_manager.go:414] "containerMap: removing container" podUID="b0716cec-3e99-4ef4-ad57-61dc8bfc9c75" containerName="revoke-keys" Oct 30 13:30:10.273597 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.273594 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0716cec-3e99-4ef4-ad57-61dc8bfc9c75" containerName="revoke-keys" Oct 30 13:30:10.273821 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.273770 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0716cec-3e99-4ef4-ad57-61dc8bfc9c75" containerName="revoke-keys" Oct 30 13:30:10.279671 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.279646 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h"] Oct 30 13:30:10.279784 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.279762 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" Oct 30 13:30:10.292124 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.291924 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" Oct 30 13:30:10.296717 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.296687 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl"] Oct 30 13:30:10.303347 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.303313 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h"] Oct 30 13:30:10.303586 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.303558 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcg22\" (UniqueName: \"kubernetes.io/projected/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-kube-api-access-dcg22\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" Oct 30 13:30:10.303728 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.303698 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-istio-data\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" Oct 30 13:30:10.303817 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.303756 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-credential-socket\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" Oct 30 13:30:10.303817 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.303798 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-istio-envoy\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" Oct 30 13:30:10.304056 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.303869 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-workload-socket\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" Oct 30 13:30:10.304182 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.304158 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-istiod-ca-cert\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" Oct 30 13:30:10.304255 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.304225 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-workload-certs\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" Oct 30 13:30:10.304344 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.304290 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-istio-podinfo\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" Oct 30 13:30:10.304344 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.304322 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-istio-token\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" Oct 30 13:30:10.348276 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.348239 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-1ba901-openshift-default-dockercfg-fl7pd\"" Oct 30 13:30:10.405761 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.405722 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/721beeba-d88a-4b37-b2a0-191c954a008d-istiod-ca-cert\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" Oct 30 13:30:10.405957 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.405789 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-credential-socket\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" Oct 30 13:30:10.405957 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.405824 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-istio-envoy\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" Oct 30 13:30:10.405957 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.405916 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6w6m\" (UniqueName: \"kubernetes.io/projected/721beeba-d88a-4b37-b2a0-191c954a008d-kube-api-access-v6w6m\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" Oct 30 13:30:10.405957 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.405952 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-workload-socket\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" Oct 30 13:30:10.406189 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.405987 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/721beeba-d88a-4b37-b2a0-191c954a008d-credential-socket\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" Oct 30 13:30:10.406189 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.406081 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/721beeba-d88a-4b37-b2a0-191c954a008d-istio-token\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" Oct 30 13:30:10.406189 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.406151 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/721beeba-d88a-4b37-b2a0-191c954a008d-workload-certs\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" Oct 30 13:30:10.406189 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.406187 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-istiod-ca-cert\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" Oct 30 13:30:10.406466 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.406235 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-credential-socket\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" Oct 30 13:30:10.406466 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.406248 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-workload-certs\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" Oct 30 13:30:10.406466 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.406326 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-workload-socket\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" Oct 30 13:30:10.406466 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.406334 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-istio-podinfo\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" Oct 30 13:30:10.406466 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.406380 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/721beeba-d88a-4b37-b2a0-191c954a008d-istio-data\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" Oct 30 13:30:10.406466 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.406405 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-istio-token\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" Oct 30 13:30:10.406788 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.406492 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/721beeba-d88a-4b37-b2a0-191c954a008d-istio-podinfo\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" Oct 30 13:30:10.406788 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.406586 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-workload-certs\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" Oct 30 13:30:10.406788 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.406627 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/721beeba-d88a-4b37-b2a0-191c954a008d-istio-envoy\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" Oct 30 13:30:10.406788 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.406692 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcg22\" (UniqueName: \"kubernetes.io/projected/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-kube-api-access-dcg22\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" Oct 30 13:30:10.406788 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.406774 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/721beeba-d88a-4b37-b2a0-191c954a008d-workload-socket\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" Oct 30 13:30:10.407001 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.406813 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-istio-data\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" Oct 30 13:30:10.407001 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.406897 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-istiod-ca-cert\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" Oct 30 13:30:10.407165 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.407148 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-istio-data\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" Oct 30 13:30:10.408759 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.408732 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-istio-envoy\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" Oct 30 13:30:10.409114 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.409095 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-istio-podinfo\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" Oct 30 13:30:10.431101 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.431074 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcg22\" (UniqueName: \"kubernetes.io/projected/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-kube-api-access-dcg22\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" Oct 30 13:30:10.431393 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.431374 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-istio-token\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" Oct 30 13:30:10.507711 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.507676 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/721beeba-d88a-4b37-b2a0-191c954a008d-istio-data\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" Oct 30 13:30:10.507898 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.507736 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/721beeba-d88a-4b37-b2a0-191c954a008d-istio-podinfo\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" Oct 30 13:30:10.507898 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.507762 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/721beeba-d88a-4b37-b2a0-191c954a008d-istio-envoy\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" Oct 30 13:30:10.507898 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.507856 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/721beeba-d88a-4b37-b2a0-191c954a008d-workload-socket\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" Oct 30 13:30:10.507898 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.507899 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/721beeba-d88a-4b37-b2a0-191c954a008d-istiod-ca-cert\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" Oct 30 13:30:10.508196 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.507969 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6w6m\" (UniqueName: \"kubernetes.io/projected/721beeba-d88a-4b37-b2a0-191c954a008d-kube-api-access-v6w6m\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" Oct 30 13:30:10.508196 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.508016 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/721beeba-d88a-4b37-b2a0-191c954a008d-credential-socket\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" Oct 30 13:30:10.508196 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.508058 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/721beeba-d88a-4b37-b2a0-191c954a008d-istio-data\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" Oct 30 13:30:10.508196 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.508080 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/721beeba-d88a-4b37-b2a0-191c954a008d-istio-token\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" Oct 30 13:30:10.508196 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.508129 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/721beeba-d88a-4b37-b2a0-191c954a008d-workload-certs\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" Oct 30 13:30:10.508196 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.508172 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/721beeba-d88a-4b37-b2a0-191c954a008d-workload-socket\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" Oct 30 13:30:10.508514 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.508347 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/721beeba-d88a-4b37-b2a0-191c954a008d-credential-socket\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" Oct 30 13:30:10.508567 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.508511 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/721beeba-d88a-4b37-b2a0-191c954a008d-workload-certs\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" Oct 30 13:30:10.508662 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.508643 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/721beeba-d88a-4b37-b2a0-191c954a008d-istiod-ca-cert\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" Oct 30 13:30:10.510159 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.510135 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/721beeba-d88a-4b37-b2a0-191c954a008d-istio-envoy\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" Oct 30 13:30:10.510447 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.510427 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/721beeba-d88a-4b37-b2a0-191c954a008d-istio-podinfo\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" Oct 30 13:30:10.534220 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.534160 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/721beeba-d88a-4b37-b2a0-191c954a008d-istio-token\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" Oct 30 13:30:10.534459 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.534441 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6w6m\" (UniqueName: \"kubernetes.io/projected/721beeba-d88a-4b37-b2a0-191c954a008d-kube-api-access-v6w6m\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" Oct 30 13:30:10.609389 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.609358 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" Oct 30 13:30:10.613638 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.613611 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" Oct 30 13:30:10.981856 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.981827 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h"] Oct 30 13:30:10.984933 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:30:10.984908 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod721beeba_d88a_4b37_b2a0_191c954a008d.slice/crio-63b34a81dcd50c04ddbfd90617c8d21609985ed01e90d40f28c8f46167667e6a WatchSource:0}: Error finding container 63b34a81dcd50c04ddbfd90617c8d21609985ed01e90d40f28c8f46167667e6a: Status 404 returned error can't find the container with id 63b34a81dcd50c04ddbfd90617c8d21609985ed01e90d40f28c8f46167667e6a Oct 30 13:30:10.987930 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.987896 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:30:10.988007 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.987983 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:30:10.988042 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.988018 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:30:10.995588 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:10.995567 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl"] Oct 30 13:30:11.000315 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:30:11.000288 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0055d65c_ebe0_4c48_964f_36a4c7e8e59a.slice/crio-efacfc5f681721aec5f25ae008b914a88075b57bd576da5cdc2888dce6c15772 WatchSource:0}: Error finding container efacfc5f681721aec5f25ae008b914a88075b57bd576da5cdc2888dce6c15772: Status 404 returned error can't find the container with id efacfc5f681721aec5f25ae008b914a88075b57bd576da5cdc2888dce6c15772 Oct 30 13:30:11.002976 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:11.002944 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:30:11.003057 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:11.003010 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:30:11.003057 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:11.003044 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:30:11.495397 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:11.495352 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" event={"ID":"721beeba-d88a-4b37-b2a0-191c954a008d","Type":"ContainerStarted","Data":"7609a223a613c24eb8b3909536a8c582f7194c689837e02a53050725620409fe"} Oct 30 13:30:11.495915 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:11.495414 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" event={"ID":"721beeba-d88a-4b37-b2a0-191c954a008d","Type":"ContainerStarted","Data":"63b34a81dcd50c04ddbfd90617c8d21609985ed01e90d40f28c8f46167667e6a"} Oct 30 13:30:11.497847 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:11.497808 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" event={"ID":"0055d65c-ebe0-4c48-964f-36a4c7e8e59a","Type":"ContainerStarted","Data":"2442b59de64ba0280a4b9f3e192ccb71926dd14625e7c9d7720f455650f7003d"} Oct 30 13:30:11.498003 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:11.497872 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" event={"ID":"0055d65c-ebe0-4c48-964f-36a4c7e8e59a","Type":"ContainerStarted","Data":"efacfc5f681721aec5f25ae008b914a88075b57bd576da5cdc2888dce6c15772"} Oct 30 13:30:11.555113 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:11.555058 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" podStartSLOduration=1.555043139 podStartE2EDuration="1.555043139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:11.546378496 +0000 UTC m=+2710.149876953" watchObservedRunningTime="2026-10-30 13:30:11.555043139 +0000 UTC m=+2710.158541582" Oct 30 13:30:11.610153 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:11.610119 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" Oct 30 13:30:11.613812 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:11.613784 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" Oct 30 13:30:11.621048 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:11.621002 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" podStartSLOduration=1.620985095 podStartE2EDuration="1.620985095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:11.617613159 +0000 UTC m=+2710.221111605" watchObservedRunningTime="2026-10-30 13:30:11.620985095 +0000 UTC m=+2710.224483541" Oct 30 13:30:12.610987 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:12.610948 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 13:30:12.611425 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:12.611008 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" podUID="0055d65c-ebe0-4c48-964f-36a4c7e8e59a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 13:30:12.615471 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:12.615442 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 13:30:12.615646 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:12.615491 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" podUID="721beeba-d88a-4b37-b2a0-191c954a008d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 13:30:13.619288 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:13.615257 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 13:30:13.699296 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:13.663534 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" podUID="0055d65c-ebe0-4c48-964f-36a4c7e8e59a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 13:30:13.699296 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:13.634906 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 13:30:13.699296 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:13.664108 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" podUID="721beeba-d88a-4b37-b2a0-191c954a008d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 13:30:14.610711 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:14.610675 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 13:30:14.610934 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:14.610728 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" podUID="0055d65c-ebe0-4c48-964f-36a4c7e8e59a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 13:30:14.614482 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:14.614419 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 13:30:14.614683 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:14.614659 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" podUID="721beeba-d88a-4b37-b2a0-191c954a008d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 13:30:15.054351 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:15.054314 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cc9ad88-nc5m4"] Oct 30 13:30:15.088800 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:15.088538 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cc9ad88-nc5m4"] Oct 30 13:30:15.515679 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:15.515644 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e946299e-2bd0-4ad8-af2c-6fc3340fd967" path="/var/lib/kubelet/pods/e946299e-2bd0-4ad8-af2c-6fc3340fd967/volumes" Oct 30 13:30:15.610662 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:15.610621 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 13:30:15.610819 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:15.610682 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" podUID="0055d65c-ebe0-4c48-964f-36a4c7e8e59a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 13:30:15.614674 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:15.614646 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 13:30:15.614814 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:15.614687 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" podUID="721beeba-d88a-4b37-b2a0-191c954a008d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 13:30:16.610390 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:16.610322 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 13:30:16.610934 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:16.610409 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" podUID="0055d65c-ebe0-4c48-964f-36a4c7e8e59a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 13:30:16.614540 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:16.614509 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 13:30:16.614700 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:16.614552 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" podUID="721beeba-d88a-4b37-b2a0-191c954a008d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 13:30:17.505838 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:17.505802 4931 scope.go:122] "RemoveContainer" containerID="584c9a6962fa63ac7b274e0ffd328569257286cbe4c42289e7822daca55b9096" Oct 30 13:30:17.610636 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:17.610595 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 13:30:17.611134 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:17.610656 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" podUID="0055d65c-ebe0-4c48-964f-36a4c7e8e59a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 13:30:17.614315 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:17.614063 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 13:30:17.614315 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:17.614113 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" podUID="721beeba-d88a-4b37-b2a0-191c954a008d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 13:30:18.506255 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:18.506220 4931 scope.go:122] "RemoveContainer" containerID="bbe0d45e37d5b0a91811f7561b098ac131672db1eeb7f8ac49202fdb44fab745" Oct 30 13:30:18.507030 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:18.507003 4931 scope.go:122] "RemoveContainer" containerID="11f0e27327c96aad37aa513810967da717e117b60d2c8fb63c4b5d1194fd5b4f" Oct 30 13:30:18.507217 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:18.507134 4931 scope.go:122] "RemoveContainer" containerID="fe52f278f1292402dd28b5be8de766a2b3724296eefee34b317a221867bd3d41" Oct 30 13:30:18.507684 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:18.507256 4931 scope.go:122] "RemoveContainer" containerID="2ec30021a01f3e7b0f3cee253bbc3ec41ff78d8daae2562a3e8fd6eb0924f7d5" Oct 30 13:30:18.507684 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:18.507337 4931 scope.go:122] "RemoveContainer" containerID="8bd1bc09525560afc1fac5ef85feba7a27d5be5822f13753d3a44edc46245def" Oct 30 13:30:18.507903 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:18.507880 4931 scope.go:122] "RemoveContainer" containerID="7dfea41cb9bf6049e289ac1826948e9ae06edadf9bd296468138a466904a6bb9" Oct 30 13:30:18.508621 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:18.508598 4931 scope.go:122] "RemoveContainer" containerID="3a9f43f02d38aab7831805fcf5ed75451d8aeada6df9b18681b236aa6242787f" Oct 30 13:30:18.618071 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:18.618043 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" Oct 30 13:30:18.619846 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:18.619682 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" Oct 30 13:30:18.620130 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:18.620106 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" Oct 30 13:30:18.620718 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:18.620698 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" Oct 30 13:30:18.620823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:18.620746 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" Oct 30 13:30:18.621364 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:18.621343 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" Oct 30 13:30:18.821126 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:18.821079 4931 generic.go:356] "Generic (PLEG): container finished" podID="122219fe-8078-42d9-b4ed-7c0333b9013c" containerID="9a4e5c4492249cc6aa0dd8cb084586e1953df66bed6aca24b59b09bd2a6354a4" exitCode=60 Oct 30 13:30:18.822384 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:18.822354 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t" event={"ID":"122219fe-8078-42d9-b4ed-7c0333b9013c","Type":"ContainerDied","Data":"9a4e5c4492249cc6aa0dd8cb084586e1953df66bed6aca24b59b09bd2a6354a4"} Oct 30 13:30:18.822594 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:18.822578 4931 scope.go:122] "RemoveContainer" containerID="584c9a6962fa63ac7b274e0ffd328569257286cbe4c42289e7822daca55b9096" Oct 30 13:30:18.823143 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:18.823125 4931 scope.go:122] "RemoveContainer" containerID="9a4e5c4492249cc6aa0dd8cb084586e1953df66bed6aca24b59b09bd2a6354a4" Oct 30 13:30:18.825175 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:30:18.825147 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t_odh-ai-gateway-infra(122219fe-8078-42d9-b4ed-7c0333b9013c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t" podUID="122219fe-8078-42d9-b4ed-7c0333b9013c" Oct 30 13:30:19.508571 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:19.507608 4931 scope.go:122] "RemoveContainer" containerID="8002b5f56b7061f96ec1e7ae7f3d11f43ba5bc56b367f09052036a368a1e204d" Oct 30 13:30:19.837617 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:19.837576 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-bnskt" event={"ID":"f23416bb-8d3b-422e-9880-fca313dfd70c","Type":"ContainerStarted","Data":"1fc5c1198c14e50586525cab8a099652cb86be816c5134999118b411f47a08fe"} Oct 30 13:30:19.840428 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:19.840399 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq" event={"ID":"32712e88-00fd-4f97-882a-a633da6a16df","Type":"ContainerStarted","Data":"ceb4092b007ed82f20a9012a78336fad1ee4f5254a2fcb0c4bd3557c8a86b6c2"} Oct 30 13:30:19.842724 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:19.842700 4931 generic.go:356] "Generic (PLEG): container finished" podID="a3d54ed7-7658-49d1-95e4-2e522546e8b3" containerID="cb5edd140903f34aa0e38a37e7369cd57f4535ff967e3a9c8d891838b8791afd" exitCode=60 Oct 30 13:30:19.842878 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:19.842766 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889450-7dzx8" event={"ID":"a3d54ed7-7658-49d1-95e4-2e522546e8b3","Type":"ContainerDied","Data":"cb5edd140903f34aa0e38a37e7369cd57f4535ff967e3a9c8d891838b8791afd"} Oct 30 13:30:19.843005 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:19.842914 4931 scope.go:122] "RemoveContainer" containerID="3a9f43f02d38aab7831805fcf5ed75451d8aeada6df9b18681b236aa6242787f" Oct 30 13:30:19.843342 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:19.843322 4931 scope.go:122] "RemoveContainer" containerID="cb5edd140903f34aa0e38a37e7369cd57f4535ff967e3a9c8d891838b8791afd" Oct 30 13:30:19.843698 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:30:19.843673 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-09db16-29889450-7dzx8_odh-ai-gateway-infra(a3d54ed7-7658-49d1-95e4-2e522546e8b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889450-7dzx8" podUID="a3d54ed7-7658-49d1-95e4-2e522546e8b3" Oct 30 13:30:19.847601 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:19.847576 4931 generic.go:356] "Generic (PLEG): container finished" podID="07b99b5b-b3b4-4621-9382-4db0db4bf317" containerID="5890d5235de80d438a5241a764ad547a2d95d3e9872271b389a29a1ccd70010f" exitCode=60 Oct 30 13:30:19.847710 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:19.847651 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ba901-29889450-fs87w" event={"ID":"07b99b5b-b3b4-4621-9382-4db0db4bf317","Type":"ContainerDied","Data":"5890d5235de80d438a5241a764ad547a2d95d3e9872271b389a29a1ccd70010f"} Oct 30 13:30:19.848007 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:19.847988 4931 scope.go:122] "RemoveContainer" containerID="5890d5235de80d438a5241a764ad547a2d95d3e9872271b389a29a1ccd70010f" Oct 30 13:30:19.848313 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:30:19.848250 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-1ba901-29889450-fs87w_odh-ai-gateway-infra(07b99b5b-b3b4-4621-9382-4db0db4bf317)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ba901-29889450-fs87w" podUID="07b99b5b-b3b4-4621-9382-4db0db4bf317" Oct 30 13:30:19.849832 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:19.849809 4931 generic.go:356] "Generic (PLEG): container finished" podID="ae4551d9-3f2a-4278-b567-243121f8e1ff" containerID="1da996228b248522aaceb725c8af40b8d5330ed2c9a59098b062e0c454e64f7e" exitCode=60 Oct 30 13:30:19.849959 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:19.849875 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd" event={"ID":"ae4551d9-3f2a-4278-b567-243121f8e1ff","Type":"ContainerDied","Data":"1da996228b248522aaceb725c8af40b8d5330ed2c9a59098b062e0c454e64f7e"} Oct 30 13:30:19.850253 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:19.850233 4931 scope.go:122] "RemoveContainer" containerID="1da996228b248522aaceb725c8af40b8d5330ed2c9a59098b062e0c454e64f7e" Oct 30 13:30:19.850535 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:30:19.850508 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd_odh-ai-gateway-infra(ae4551d9-3f2a-4278-b567-243121f8e1ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd" podUID="ae4551d9-3f2a-4278-b567-243121f8e1ff" Oct 30 13:30:19.852677 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:19.852658 4931 generic.go:356] "Generic (PLEG): container finished" podID="65f88ff2-f7d4-4207-b2d1-a49a6f052a2c" containerID="e275a4a1bca2d87333f8f493566adf17850fded013dc8448216cf0c750bc3ff5" exitCode=60 Oct 30 13:30:19.852790 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:19.852767 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg" event={"ID":"65f88ff2-f7d4-4207-b2d1-a49a6f052a2c","Type":"ContainerDied","Data":"e275a4a1bca2d87333f8f493566adf17850fded013dc8448216cf0c750bc3ff5"} Oct 30 13:30:19.853160 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:19.853138 4931 scope.go:122] "RemoveContainer" containerID="e275a4a1bca2d87333f8f493566adf17850fded013dc8448216cf0c750bc3ff5" Oct 30 13:30:19.853472 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:30:19.853444 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg_odh-ai-gateway-infra(65f88ff2-f7d4-4207-b2d1-a49a6f052a2c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg" podUID="65f88ff2-f7d4-4207-b2d1-a49a6f052a2c" Oct 30 13:30:19.854909 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:19.854886 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb" event={"ID":"52d65cd3-640f-46e7-90e9-534c3c990930","Type":"ContainerStarted","Data":"7ec127db9c7ab49b6535ab5c7266d44342ef3729b0bbd324f0ff44221b8a1378"} Oct 30 13:30:19.855517 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:19.855501 4931 scope.go:122] "RemoveContainer" containerID="7ec127db9c7ab49b6535ab5c7266d44342ef3729b0bbd324f0ff44221b8a1378" Oct 30 13:30:19.855741 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:30:19.855723 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb_odh-ai-gateway-infra(52d65cd3-640f-46e7-90e9-534c3c990930)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb" podUID="52d65cd3-640f-46e7-90e9-534c3c990930" Oct 30 13:30:19.900673 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:19.900643 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t"] Oct 30 13:30:20.146918 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:20.146884 4931 scope.go:122] "RemoveContainer" containerID="7dfea41cb9bf6049e289ac1826948e9ae06edadf9bd296468138a466904a6bb9" Oct 30 13:30:20.156199 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:20.156167 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t" Oct 30 13:30:20.162510 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:20.162481 4931 scope.go:122] "RemoveContainer" containerID="bbe0d45e37d5b0a91811f7561b098ac131672db1eeb7f8ac49202fdb44fab745" Oct 30 13:30:20.180213 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:20.180186 4931 scope.go:122] "RemoveContainer" containerID="fe52f278f1292402dd28b5be8de766a2b3724296eefee34b317a221867bd3d41" Oct 30 13:30:20.303636 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:20.303545 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/122219fe-8078-42d9-b4ed-7c0333b9013c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/122219fe-8078-42d9-b4ed-7c0333b9013c-maas-api-service-ca\") pod \"122219fe-8078-42d9-b4ed-7c0333b9013c\" (UID: \"122219fe-8078-42d9-b4ed-7c0333b9013c\") " Oct 30 13:30:20.303820 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:20.303716 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/122219fe-8078-42d9-b4ed-7c0333b9013c-kube-api-access-lmddv\" (UniqueName: \"kubernetes.io/projected/122219fe-8078-42d9-b4ed-7c0333b9013c-kube-api-access-lmddv\") pod \"122219fe-8078-42d9-b4ed-7c0333b9013c\" (UID: \"122219fe-8078-42d9-b4ed-7c0333b9013c\") " Oct 30 13:30:20.303921 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:20.303891 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/122219fe-8078-42d9-b4ed-7c0333b9013c-maas-api-service-ca" pod "122219fe-8078-42d9-b4ed-7c0333b9013c" (UID: "122219fe-8078-42d9-b4ed-7c0333b9013c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:20.304533 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:20.304489 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/122219fe-8078-42d9-b4ed-7c0333b9013c-maas-api-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:30:20.306686 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:20.306660 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/122219fe-8078-42d9-b4ed-7c0333b9013c-kube-api-access-lmddv" pod "122219fe-8078-42d9-b4ed-7c0333b9013c" (UID: "122219fe-8078-42d9-b4ed-7c0333b9013c"). InnerVolumeSpecName "kube-api-access-lmddv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:20.405898 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:20.405851 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmddv\" (UniqueName: \"kubernetes.io/projected/122219fe-8078-42d9-b4ed-7c0333b9013c-kube-api-access-lmddv\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:30:20.506467 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:20.506429 4931 scope.go:122] "RemoveContainer" containerID="ebb797d61af0ecfd0faf4e35994322dbce96f9b7b044ab736aba22315af423c6" Oct 30 13:30:20.867610 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:20.867576 4931 generic.go:356] "Generic (PLEG): container finished" podID="52d65cd3-640f-46e7-90e9-534c3c990930" containerID="7ec127db9c7ab49b6535ab5c7266d44342ef3729b0bbd324f0ff44221b8a1378" exitCode=60 Oct 30 13:30:20.868046 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:20.867650 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb" event={"ID":"52d65cd3-640f-46e7-90e9-534c3c990930","Type":"ContainerDied","Data":"7ec127db9c7ab49b6535ab5c7266d44342ef3729b0bbd324f0ff44221b8a1378"} Oct 30 13:30:20.868046 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:20.867712 4931 scope.go:122] "RemoveContainer" containerID="2ec30021a01f3e7b0f3cee253bbc3ec41ff78d8daae2562a3e8fd6eb0924f7d5" Oct 30 13:30:20.868218 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:20.868191 4931 scope.go:122] "RemoveContainer" containerID="7ec127db9c7ab49b6535ab5c7266d44342ef3729b0bbd324f0ff44221b8a1378" Oct 30 13:30:20.868640 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:30:20.868610 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb_odh-ai-gateway-infra(52d65cd3-640f-46e7-90e9-534c3c990930)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb" podUID="52d65cd3-640f-46e7-90e9-534c3c990930" Oct 30 13:30:20.869938 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:20.869918 4931 generic.go:356] "Generic (PLEG): container finished" podID="f23416bb-8d3b-422e-9880-fca313dfd70c" containerID="1fc5c1198c14e50586525cab8a099652cb86be816c5134999118b411f47a08fe" exitCode=60 Oct 30 13:30:20.870031 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:20.869989 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-bnskt" event={"ID":"f23416bb-8d3b-422e-9880-fca313dfd70c","Type":"ContainerDied","Data":"1fc5c1198c14e50586525cab8a099652cb86be816c5134999118b411f47a08fe"} Oct 30 13:30:20.870397 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:20.870378 4931 scope.go:122] "RemoveContainer" containerID="1fc5c1198c14e50586525cab8a099652cb86be816c5134999118b411f47a08fe" Oct 30 13:30:20.870792 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:30:20.870726 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889450-bnskt_odh-ai-gateway-infra(f23416bb-8d3b-422e-9880-fca313dfd70c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-bnskt" podUID="f23416bb-8d3b-422e-9880-fca313dfd70c" Oct 30 13:30:20.872008 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:20.871990 4931 generic.go:356] "Generic (PLEG): container finished" podID="32712e88-00fd-4f97-882a-a633da6a16df" containerID="ceb4092b007ed82f20a9012a78336fad1ee4f5254a2fcb0c4bd3557c8a86b6c2" exitCode=60 Oct 30 13:30:20.872101 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:20.872062 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq" event={"ID":"32712e88-00fd-4f97-882a-a633da6a16df","Type":"ContainerDied","Data":"ceb4092b007ed82f20a9012a78336fad1ee4f5254a2fcb0c4bd3557c8a86b6c2"} Oct 30 13:30:20.872412 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:20.872393 4931 scope.go:122] "RemoveContainer" containerID="ceb4092b007ed82f20a9012a78336fad1ee4f5254a2fcb0c4bd3557c8a86b6c2" Oct 30 13:30:20.872674 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:30:20.872654 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq_odh-ai-gateway-infra(32712e88-00fd-4f97-882a-a633da6a16df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq" podUID="32712e88-00fd-4f97-882a-a633da6a16df" Oct 30 13:30:20.875029 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:20.875013 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t" Oct 30 13:30:20.875029 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:20.875019 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t" event={"ID":"122219fe-8078-42d9-b4ed-7c0333b9013c","Type":"ContainerDied","Data":"78d1e3688c874a2c96211314709ea9482a790d094ee79340f3dba57c4b33da41"} Oct 30 13:30:20.880315 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:20.880292 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z" event={"ID":"a5592ca1-ccf3-4297-bc42-ea5aa0cca97a","Type":"ContainerStarted","Data":"4d5b01b6310cc7b76db9ccdd0eb8449558be2d21b860c07fab7ee6ea774e24f9"} Oct 30 13:30:20.891492 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:20.891470 4931 scope.go:122] "RemoveContainer" containerID="8bd1bc09525560afc1fac5ef85feba7a27d5be5822f13753d3a44edc46245def" Oct 30 13:30:20.896014 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:20.895989 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889450-bnskt"] Oct 30 13:30:20.906480 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:20.906452 4931 scope.go:122] "RemoveContainer" containerID="11f0e27327c96aad37aa513810967da717e117b60d2c8fb63c4b5d1194fd5b4f" Oct 30 13:30:20.921992 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:20.921967 4931 scope.go:122] "RemoveContainer" containerID="9a4e5c4492249cc6aa0dd8cb084586e1953df66bed6aca24b59b09bd2a6354a4" Oct 30 13:30:20.950717 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:20.950694 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg"] Oct 30 13:30:20.995784 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:20.995752 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq"] Oct 30 13:30:21.047352 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.047318 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb"] Oct 30 13:30:21.068330 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.068296 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t"] Oct 30 13:30:21.083982 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.083953 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889450-tqg4t"] Oct 30 13:30:21.093973 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.093948 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889450-7dzx8"] Oct 30 13:30:21.130483 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.130451 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg" Oct 30 13:30:21.130629 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.130610 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd"] Oct 30 13:30:21.240332 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.225760 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65f88ff2-f7d4-4207-b2d1-a49a6f052a2c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f88ff2-f7d4-4207-b2d1-a49a6f052a2c-maas-api-service-ca\") pod \"65f88ff2-f7d4-4207-b2d1-a49a6f052a2c\" (UID: \"65f88ff2-f7d4-4207-b2d1-a49a6f052a2c\") " Oct 30 13:30:21.240332 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.225808 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65f88ff2-f7d4-4207-b2d1-a49a6f052a2c-kube-api-access-hxm7r\" (UniqueName: \"kubernetes.io/projected/65f88ff2-f7d4-4207-b2d1-a49a6f052a2c-kube-api-access-hxm7r\") pod \"65f88ff2-f7d4-4207-b2d1-a49a6f052a2c\" (UID: \"65f88ff2-f7d4-4207-b2d1-a49a6f052a2c\") " Oct 30 13:30:21.240332 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.239652 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f88ff2-f7d4-4207-b2d1-a49a6f052a2c-maas-api-service-ca" pod "65f88ff2-f7d4-4207-b2d1-a49a6f052a2c" (UID: "65f88ff2-f7d4-4207-b2d1-a49a6f052a2c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:21.251511 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.243077 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f88ff2-f7d4-4207-b2d1-a49a6f052a2c-kube-api-access-hxm7r" pod "65f88ff2-f7d4-4207-b2d1-a49a6f052a2c" (UID: "65f88ff2-f7d4-4207-b2d1-a49a6f052a2c"). InnerVolumeSpecName "kube-api-access-hxm7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:21.261191 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.260511 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ba901-29889450-fs87w"] Oct 30 13:30:21.320227 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.320139 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889450-7dzx8" Oct 30 13:30:21.330148 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.330050 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f88ff2-f7d4-4207-b2d1-a49a6f052a2c-maas-api-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:30:21.330148 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.330081 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxm7r\" (UniqueName: \"kubernetes.io/projected/65f88ff2-f7d4-4207-b2d1-a49a6f052a2c-kube-api-access-hxm7r\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:30:21.431948 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.431910 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3d54ed7-7658-49d1-95e4-2e522546e8b3-kube-api-access-w9kbm\" (UniqueName: \"kubernetes.io/projected/a3d54ed7-7658-49d1-95e4-2e522546e8b3-kube-api-access-w9kbm\") pod \"a3d54ed7-7658-49d1-95e4-2e522546e8b3\" (UID: \"a3d54ed7-7658-49d1-95e4-2e522546e8b3\") " Oct 30 13:30:21.432406 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.432212 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3d54ed7-7658-49d1-95e4-2e522546e8b3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3d54ed7-7658-49d1-95e4-2e522546e8b3-maas-api-service-ca\") pod \"a3d54ed7-7658-49d1-95e4-2e522546e8b3\" (UID: \"a3d54ed7-7658-49d1-95e4-2e522546e8b3\") " Oct 30 13:30:21.436308 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.436241 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3d54ed7-7658-49d1-95e4-2e522546e8b3-kube-api-access-w9kbm" pod "a3d54ed7-7658-49d1-95e4-2e522546e8b3" (UID: "a3d54ed7-7658-49d1-95e4-2e522546e8b3"). InnerVolumeSpecName "kube-api-access-w9kbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:21.437838 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.437808 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3d54ed7-7658-49d1-95e4-2e522546e8b3-maas-api-service-ca" pod "a3d54ed7-7658-49d1-95e4-2e522546e8b3" (UID: "a3d54ed7-7658-49d1-95e4-2e522546e8b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:21.462413 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.462389 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd" Oct 30 13:30:21.498905 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.498880 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ba901-29889450-fs87w" Oct 30 13:30:21.516450 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.516421 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="122219fe-8078-42d9-b4ed-7c0333b9013c" path="/var/lib/kubelet/pods/122219fe-8078-42d9-b4ed-7c0333b9013c/volumes" Oct 30 13:30:21.538656 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.538612 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae4551d9-3f2a-4278-b567-243121f8e1ff-kube-api-access-22fwv\" (UniqueName: \"kubernetes.io/projected/ae4551d9-3f2a-4278-b567-243121f8e1ff-kube-api-access-22fwv\") pod \"ae4551d9-3f2a-4278-b567-243121f8e1ff\" (UID: \"ae4551d9-3f2a-4278-b567-243121f8e1ff\") " Oct 30 13:30:21.538777 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.538753 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae4551d9-3f2a-4278-b567-243121f8e1ff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae4551d9-3f2a-4278-b567-243121f8e1ff-maas-api-service-ca\") pod \"ae4551d9-3f2a-4278-b567-243121f8e1ff\" (UID: \"ae4551d9-3f2a-4278-b567-243121f8e1ff\") " Oct 30 13:30:21.539459 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.539430 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae4551d9-3f2a-4278-b567-243121f8e1ff-maas-api-service-ca" pod "ae4551d9-3f2a-4278-b567-243121f8e1ff" (UID: "ae4551d9-3f2a-4278-b567-243121f8e1ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:21.539694 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.539671 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae4551d9-3f2a-4278-b567-243121f8e1ff-maas-api-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:30:21.539753 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.539701 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3d54ed7-7658-49d1-95e4-2e522546e8b3-maas-api-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:30:21.539753 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.539716 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9kbm\" (UniqueName: \"kubernetes.io/projected/a3d54ed7-7658-49d1-95e4-2e522546e8b3-kube-api-access-w9kbm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:30:21.542863 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.542831 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae4551d9-3f2a-4278-b567-243121f8e1ff-kube-api-access-22fwv" pod "ae4551d9-3f2a-4278-b567-243121f8e1ff" (UID: "ae4551d9-3f2a-4278-b567-243121f8e1ff"). InnerVolumeSpecName "kube-api-access-22fwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:21.640393 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.640303 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07b99b5b-b3b4-4621-9382-4db0db4bf317-kube-api-access-qtp7g\" (UniqueName: \"kubernetes.io/projected/07b99b5b-b3b4-4621-9382-4db0db4bf317-kube-api-access-qtp7g\") pod \"07b99b5b-b3b4-4621-9382-4db0db4bf317\" (UID: \"07b99b5b-b3b4-4621-9382-4db0db4bf317\") " Oct 30 13:30:21.640567 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.640425 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07b99b5b-b3b4-4621-9382-4db0db4bf317-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07b99b5b-b3b4-4621-9382-4db0db4bf317-maas-api-service-ca\") pod \"07b99b5b-b3b4-4621-9382-4db0db4bf317\" (UID: \"07b99b5b-b3b4-4621-9382-4db0db4bf317\") " Oct 30 13:30:21.640854 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.640823 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07b99b5b-b3b4-4621-9382-4db0db4bf317-maas-api-service-ca" pod "07b99b5b-b3b4-4621-9382-4db0db4bf317" (UID: "07b99b5b-b3b4-4621-9382-4db0db4bf317"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:21.641116 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.641097 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22fwv\" (UniqueName: \"kubernetes.io/projected/ae4551d9-3f2a-4278-b567-243121f8e1ff-kube-api-access-22fwv\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:30:21.641655 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.641117 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07b99b5b-b3b4-4621-9382-4db0db4bf317-maas-api-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:30:21.659936 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.659901 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07b99b5b-b3b4-4621-9382-4db0db4bf317-kube-api-access-qtp7g" pod "07b99b5b-b3b4-4621-9382-4db0db4bf317" (UID: "07b99b5b-b3b4-4621-9382-4db0db4bf317"). InnerVolumeSpecName "kube-api-access-qtp7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:21.743769 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.743732 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtp7g\" (UniqueName: \"kubernetes.io/projected/07b99b5b-b3b4-4621-9382-4db0db4bf317-kube-api-access-qtp7g\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:30:21.893838 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.893758 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd" Oct 30 13:30:21.893838 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.893791 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd" event={"ID":"ae4551d9-3f2a-4278-b567-243121f8e1ff","Type":"ContainerDied","Data":"61cdb499cfaf640b09a58e3df71c061477551182189b5d889da9162a9a885910"} Oct 30 13:30:21.894478 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.893882 4931 scope.go:122] "RemoveContainer" containerID="1da996228b248522aaceb725c8af40b8d5330ed2c9a59098b062e0c454e64f7e" Oct 30 13:30:21.896036 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.896012 4931 generic.go:356] "Generic (PLEG): container finished" podID="a5592ca1-ccf3-4297-bc42-ea5aa0cca97a" containerID="4d5b01b6310cc7b76db9ccdd0eb8449558be2d21b860c07fab7ee6ea774e24f9" exitCode=60 Oct 30 13:30:21.896157 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.896084 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z" event={"ID":"a5592ca1-ccf3-4297-bc42-ea5aa0cca97a","Type":"ContainerDied","Data":"4d5b01b6310cc7b76db9ccdd0eb8449558be2d21b860c07fab7ee6ea774e24f9"} Oct 30 13:30:21.896549 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.896531 4931 scope.go:122] "RemoveContainer" containerID="4d5b01b6310cc7b76db9ccdd0eb8449558be2d21b860c07fab7ee6ea774e24f9" Oct 30 13:30:21.896894 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:30:21.896871 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z_odh-ai-gateway-infra(a5592ca1-ccf3-4297-bc42-ea5aa0cca97a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z" podUID="a5592ca1-ccf3-4297-bc42-ea5aa0cca97a" Oct 30 13:30:21.897621 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.897603 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg" Oct 30 13:30:21.897723 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.897627 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg" event={"ID":"65f88ff2-f7d4-4207-b2d1-a49a6f052a2c","Type":"ContainerDied","Data":"6947ed962fa149ecd185c5a8e656e197a28e7023ab3c2feb3ea3a57dfb53aa9f"} Oct 30 13:30:21.902597 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.902575 4931 generic.go:356] "Generic (PLEG): container finished" podID="775cb414-3b30-4adb-a746-d8a9fb548b48" containerID="dd0e2e9093680611df565d35a2e423833d0b8e1523b6edf3786506ef95714f16" exitCode=60 Oct 30 13:30:21.902710 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.902661 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj" event={"ID":"775cb414-3b30-4adb-a746-d8a9fb548b48","Type":"ContainerDied","Data":"dd0e2e9093680611df565d35a2e423833d0b8e1523b6edf3786506ef95714f16"} Oct 30 13:30:21.903138 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.903118 4931 scope.go:122] "RemoveContainer" containerID="dd0e2e9093680611df565d35a2e423833d0b8e1523b6edf3786506ef95714f16" Oct 30 13:30:21.903489 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:30:21.903463 4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj_odh-ai-gateway-infra(775cb414-3b30-4adb-a746-d8a9fb548b48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj" podUID="775cb414-3b30-4adb-a746-d8a9fb548b48" Oct 30 13:30:21.908187 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.908159 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889450-7dzx8" Oct 30 13:30:21.908650 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.908623 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889450-7dzx8" event={"ID":"a3d54ed7-7658-49d1-95e4-2e522546e8b3","Type":"ContainerDied","Data":"7548da559110c9c0ac82482bbd37c71e8c0b895041fe891c6f615b27a937dcea"} Oct 30 13:30:21.909255 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.909233 4931 scope.go:122] "RemoveContainer" containerID="8002b5f56b7061f96ec1e7ae7f3d11f43ba5bc56b367f09052036a368a1e204d" Oct 30 13:30:21.912685 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.912665 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ba901-29889450-fs87w" event={"ID":"07b99b5b-b3b4-4621-9382-4db0db4bf317","Type":"ContainerDied","Data":"995f1732fcfbf7c8c17b50040ef60208001d80cd377d29130298e023b5d6b72b"} Oct 30 13:30:21.912780 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.912761 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ba901-29889450-fs87w" Oct 30 13:30:21.926411 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.926385 4931 scope.go:122] "RemoveContainer" containerID="e275a4a1bca2d87333f8f493566adf17850fded013dc8448216cf0c750bc3ff5" Oct 30 13:30:21.939830 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.939810 4931 scope.go:122] "RemoveContainer" containerID="ebb797d61af0ecfd0faf4e35994322dbce96f9b7b044ab736aba22315af423c6" Oct 30 13:30:21.961119 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.961096 4931 scope.go:122] "RemoveContainer" containerID="cb5edd140903f34aa0e38a37e7369cd57f4535ff967e3a9c8d891838b8791afd" Oct 30 13:30:21.973095 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.973003 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd"] Oct 30 13:30:21.973490 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.973472 4931 scope.go:122] "RemoveContainer" containerID="5890d5235de80d438a5241a764ad547a2d95d3e9872271b389a29a1ccd70010f" Oct 30 13:30:21.980627 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:21.980604 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-51288b-29889450-cvhqd"] Oct 30 13:30:22.042467 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.039516 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z"] Oct 30 13:30:22.068916 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.068886 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg"] Oct 30 13:30:22.074028 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.073998 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889450-kv6pg"] Oct 30 13:30:22.099292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.099171 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb" Oct 30 13:30:22.186323 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.186300 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq" Oct 30 13:30:22.193939 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.193872 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-bnskt" Oct 30 13:30:22.255929 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.255899 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52d65cd3-640f-46e7-90e9-534c3c990930-kube-api-access-z57n7\" (UniqueName: \"kubernetes.io/projected/52d65cd3-640f-46e7-90e9-534c3c990930-kube-api-access-z57n7\") pod \"52d65cd3-640f-46e7-90e9-534c3c990930\" (UID: \"52d65cd3-640f-46e7-90e9-534c3c990930\") " Oct 30 13:30:22.256114 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.255965 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f23416bb-8d3b-422e-9880-fca313dfd70c-kube-api-access-96p5j\" (UniqueName: \"kubernetes.io/projected/f23416bb-8d3b-422e-9880-fca313dfd70c-kube-api-access-96p5j\") pod \"f23416bb-8d3b-422e-9880-fca313dfd70c\" (UID: \"f23416bb-8d3b-422e-9880-fca313dfd70c\") " Oct 30 13:30:22.256185 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.256130 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32712e88-00fd-4f97-882a-a633da6a16df-kube-api-access-b4pbp\" (UniqueName: \"kubernetes.io/projected/32712e88-00fd-4f97-882a-a633da6a16df-kube-api-access-b4pbp\") pod \"32712e88-00fd-4f97-882a-a633da6a16df\" (UID: \"32712e88-00fd-4f97-882a-a633da6a16df\") " Oct 30 13:30:22.256185 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.256149 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32712e88-00fd-4f97-882a-a633da6a16df-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32712e88-00fd-4f97-882a-a633da6a16df-maas-api-service-ca\") pod \"32712e88-00fd-4f97-882a-a633da6a16df\" (UID: \"32712e88-00fd-4f97-882a-a633da6a16df\") " Oct 30 13:30:22.256185 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.256179 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52d65cd3-640f-46e7-90e9-534c3c990930-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52d65cd3-640f-46e7-90e9-534c3c990930-maas-api-service-ca\") pod \"52d65cd3-640f-46e7-90e9-534c3c990930\" (UID: \"52d65cd3-640f-46e7-90e9-534c3c990930\") " Oct 30 13:30:22.256436 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.256223 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f23416bb-8d3b-422e-9880-fca313dfd70c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f23416bb-8d3b-422e-9880-fca313dfd70c-maas-api-service-ca\") pod \"f23416bb-8d3b-422e-9880-fca313dfd70c\" (UID: \"f23416bb-8d3b-422e-9880-fca313dfd70c\") " Oct 30 13:30:22.256722 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.256616 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32712e88-00fd-4f97-882a-a633da6a16df-maas-api-service-ca" pod "32712e88-00fd-4f97-882a-a633da6a16df" (UID: "32712e88-00fd-4f97-882a-a633da6a16df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:22.256722 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.256678 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52d65cd3-640f-46e7-90e9-534c3c990930-maas-api-service-ca" pod "52d65cd3-640f-46e7-90e9-534c3c990930" (UID: "52d65cd3-640f-46e7-90e9-534c3c990930"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:22.256962 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.256768 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f23416bb-8d3b-422e-9880-fca313dfd70c-maas-api-service-ca" pod "f23416bb-8d3b-422e-9880-fca313dfd70c" (UID: "f23416bb-8d3b-422e-9880-fca313dfd70c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:22.257007 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.256985 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32712e88-00fd-4f97-882a-a633da6a16df-maas-api-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:30:22.257044 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.257008 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52d65cd3-640f-46e7-90e9-534c3c990930-maas-api-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:30:22.257044 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.257023 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f23416bb-8d3b-422e-9880-fca313dfd70c-maas-api-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:30:22.259632 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.259605 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52d65cd3-640f-46e7-90e9-534c3c990930-kube-api-access-z57n7" pod "52d65cd3-640f-46e7-90e9-534c3c990930" (UID: "52d65cd3-640f-46e7-90e9-534c3c990930"). InnerVolumeSpecName "kube-api-access-z57n7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:22.259766 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.259641 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32712e88-00fd-4f97-882a-a633da6a16df-kube-api-access-b4pbp" pod "32712e88-00fd-4f97-882a-a633da6a16df" (UID: "32712e88-00fd-4f97-882a-a633da6a16df"). InnerVolumeSpecName "kube-api-access-b4pbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:22.260047 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.260026 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f23416bb-8d3b-422e-9880-fca313dfd70c-kube-api-access-96p5j" pod "f23416bb-8d3b-422e-9880-fca313dfd70c" (UID: "f23416bb-8d3b-422e-9880-fca313dfd70c"). InnerVolumeSpecName "kube-api-access-96p5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:22.285099 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.285068 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889450-7dzx8"] Oct 30 13:30:22.294514 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.294483 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-09db16-29889450-7dzx8"] Oct 30 13:30:22.358854 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.358818 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z57n7\" (UniqueName: \"kubernetes.io/projected/52d65cd3-640f-46e7-90e9-534c3c990930-kube-api-access-z57n7\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:30:22.359026 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.358858 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96p5j\" (UniqueName: \"kubernetes.io/projected/f23416bb-8d3b-422e-9880-fca313dfd70c-kube-api-access-96p5j\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:30:22.359026 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.358875 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4pbp\" (UniqueName: \"kubernetes.io/projected/32712e88-00fd-4f97-882a-a633da6a16df-kube-api-access-b4pbp\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:30:22.377964 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.377928 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ba901-29889450-fs87w"] Oct 30 13:30:22.387733 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.387705 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ba901-29889450-fs87w"] Oct 30 13:30:22.926574 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.926546 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb" Oct 30 13:30:22.926976 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.926538 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb" event={"ID":"52d65cd3-640f-46e7-90e9-534c3c990930","Type":"ContainerDied","Data":"7a2770b089b697be63eb7ba3344384b639fb5032b98ae129e105827cf0e16487"} Oct 30 13:30:22.926976 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.926754 4931 scope.go:122] "RemoveContainer" containerID="7ec127db9c7ab49b6535ab5c7266d44342ef3729b0bbd324f0ff44221b8a1378" Oct 30 13:30:22.931483 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.931339 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-bnskt" event={"ID":"f23416bb-8d3b-422e-9880-fca313dfd70c","Type":"ContainerDied","Data":"9370dc490004a3558068a16d003bfe233d54a1f702e99f9a07fc83a738686a46"} Oct 30 13:30:22.931483 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.931426 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-bnskt" Oct 30 13:30:22.938554 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.938519 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq" event={"ID":"32712e88-00fd-4f97-882a-a633da6a16df","Type":"ContainerDied","Data":"0921449db53735b721eb7ef626a48f4c8cc569a8b64257f51d994142dd226f28"} Oct 30 13:30:22.938669 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.938531 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq" Oct 30 13:30:22.940175 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.940145 4931 scope.go:122] "RemoveContainer" containerID="1fc5c1198c14e50586525cab8a099652cb86be816c5134999118b411f47a08fe" Oct 30 13:30:22.953196 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:22.953173 4931 scope.go:122] "RemoveContainer" containerID="ceb4092b007ed82f20a9012a78336fad1ee4f5254a2fcb0c4bd3557c8a86b6c2" Oct 30 13:30:23.018332 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:23.018233 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq"] Oct 30 13:30:23.099835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:23.099800 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889450-k9kkq"] Oct 30 13:30:23.109015 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:23.108987 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb"] Oct 30 13:30:23.115739 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:23.115711 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0b2c92-29889450-4v5vb"] Oct 30 13:30:23.121589 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:23.121569 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889450-bnskt"] Oct 30 13:30:23.128262 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:23.128232 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889450-bnskt"] Oct 30 13:30:23.173474 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:23.173448 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj"] Oct 30 13:30:23.179641 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:23.179621 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z" Oct 30 13:30:23.280979 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:23.280898 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5592ca1-ccf3-4297-bc42-ea5aa0cca97a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5592ca1-ccf3-4297-bc42-ea5aa0cca97a-maas-api-service-ca\") pod \"a5592ca1-ccf3-4297-bc42-ea5aa0cca97a\" (UID: \"a5592ca1-ccf3-4297-bc42-ea5aa0cca97a\") " Oct 30 13:30:23.281248 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:23.281226 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5592ca1-ccf3-4297-bc42-ea5aa0cca97a-kube-api-access-v6f6b\" (UniqueName: \"kubernetes.io/projected/a5592ca1-ccf3-4297-bc42-ea5aa0cca97a-kube-api-access-v6f6b\") pod \"a5592ca1-ccf3-4297-bc42-ea5aa0cca97a\" (UID: \"a5592ca1-ccf3-4297-bc42-ea5aa0cca97a\") " Oct 30 13:30:23.282763 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:23.282739 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5592ca1-ccf3-4297-bc42-ea5aa0cca97a-maas-api-service-ca" pod "a5592ca1-ccf3-4297-bc42-ea5aa0cca97a" (UID: "a5592ca1-ccf3-4297-bc42-ea5aa0cca97a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:23.290308 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:23.287055 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5592ca1-ccf3-4297-bc42-ea5aa0cca97a-kube-api-access-v6f6b" pod "a5592ca1-ccf3-4297-bc42-ea5aa0cca97a" (UID: "a5592ca1-ccf3-4297-bc42-ea5aa0cca97a"). InnerVolumeSpecName "kube-api-access-v6f6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:23.341232 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:23.341208 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj" Oct 30 13:30:23.383718 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:23.383686 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5592ca1-ccf3-4297-bc42-ea5aa0cca97a-maas-api-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:30:23.383718 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:23.383717 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6f6b\" (UniqueName: \"kubernetes.io/projected/a5592ca1-ccf3-4297-bc42-ea5aa0cca97a-kube-api-access-v6f6b\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:30:23.484572 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:23.484508 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/775cb414-3b30-4adb-a746-d8a9fb548b48-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/775cb414-3b30-4adb-a746-d8a9fb548b48-maas-api-service-ca\") pod \"775cb414-3b30-4adb-a746-d8a9fb548b48\" (UID: \"775cb414-3b30-4adb-a746-d8a9fb548b48\") " Oct 30 13:30:23.484849 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:23.484831 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/775cb414-3b30-4adb-a746-d8a9fb548b48-kube-api-access-lbzcd\" (UniqueName: \"kubernetes.io/projected/775cb414-3b30-4adb-a746-d8a9fb548b48-kube-api-access-lbzcd\") pod \"775cb414-3b30-4adb-a746-d8a9fb548b48\" (UID: \"775cb414-3b30-4adb-a746-d8a9fb548b48\") " Oct 30 13:30:23.484892 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:23.484849 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/775cb414-3b30-4adb-a746-d8a9fb548b48-maas-api-service-ca" pod "775cb414-3b30-4adb-a746-d8a9fb548b48" (UID: "775cb414-3b30-4adb-a746-d8a9fb548b48"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:23.485387 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:23.485364 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/775cb414-3b30-4adb-a746-d8a9fb548b48-maas-api-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:30:23.486863 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:23.486841 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/775cb414-3b30-4adb-a746-d8a9fb548b48-kube-api-access-lbzcd" pod "775cb414-3b30-4adb-a746-d8a9fb548b48" (UID: "775cb414-3b30-4adb-a746-d8a9fb548b48"). InnerVolumeSpecName "kube-api-access-lbzcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:23.515220 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:23.515189 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07b99b5b-b3b4-4621-9382-4db0db4bf317" path="/var/lib/kubelet/pods/07b99b5b-b3b4-4621-9382-4db0db4bf317/volumes" Oct 30 13:30:23.515669 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:23.515654 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32712e88-00fd-4f97-882a-a633da6a16df" path="/var/lib/kubelet/pods/32712e88-00fd-4f97-882a-a633da6a16df/volumes" Oct 30 13:30:23.516048 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:23.516036 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52d65cd3-640f-46e7-90e9-534c3c990930" path="/var/lib/kubelet/pods/52d65cd3-640f-46e7-90e9-534c3c990930/volumes" Oct 30 13:30:23.516698 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:23.516681 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65f88ff2-f7d4-4207-b2d1-a49a6f052a2c" path="/var/lib/kubelet/pods/65f88ff2-f7d4-4207-b2d1-a49a6f052a2c/volumes" Oct 30 13:30:23.517102 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:23.517089 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3d54ed7-7658-49d1-95e4-2e522546e8b3" path="/var/lib/kubelet/pods/a3d54ed7-7658-49d1-95e4-2e522546e8b3/volumes" Oct 30 13:30:23.517537 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:23.517522 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae4551d9-3f2a-4278-b567-243121f8e1ff" path="/var/lib/kubelet/pods/ae4551d9-3f2a-4278-b567-243121f8e1ff/volumes" Oct 30 13:30:23.518205 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:23.518191 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f23416bb-8d3b-422e-9880-fca313dfd70c" path="/var/lib/kubelet/pods/f23416bb-8d3b-422e-9880-fca313dfd70c/volumes" Oct 30 13:30:23.586972 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:23.586891 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbzcd\" (UniqueName: \"kubernetes.io/projected/775cb414-3b30-4adb-a746-d8a9fb548b48-kube-api-access-lbzcd\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:30:23.951335 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:23.951299 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z" event={"ID":"a5592ca1-ccf3-4297-bc42-ea5aa0cca97a","Type":"ContainerDied","Data":"7b363efa746f9af5f4eb7966fd8c4c723a0bc906beec84947408fc737eef705d"} Oct 30 13:30:23.951335 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:23.951325 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z" Oct 30 13:30:23.951865 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:23.951359 4931 scope.go:122] "RemoveContainer" containerID="4d5b01b6310cc7b76db9ccdd0eb8449558be2d21b860c07fab7ee6ea774e24f9" Oct 30 13:30:23.954395 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:23.954372 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj" Oct 30 13:30:23.954395 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:23.954387 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj" event={"ID":"775cb414-3b30-4adb-a746-d8a9fb548b48","Type":"ContainerDied","Data":"685a4ddb43edbe2ae6cc5710d59eb693f9b3829e3588576b0a442e2728b5b06f"} Oct 30 13:30:23.962631 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:23.962614 4931 scope.go:122] "RemoveContainer" containerID="dd0e2e9093680611df565d35a2e423833d0b8e1523b6edf3786506ef95714f16" Oct 30 13:30:24.009034 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:24.009009 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z"] Oct 30 13:30:24.022014 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:24.021992 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889450-vk86z"] Oct 30 13:30:24.059192 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:24.059170 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj"] Oct 30 13:30:24.062993 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:24.062973 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c90b9e-29889450-fwldj"] Oct 30 13:30:25.514414 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:25.514382 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="775cb414-3b30-4adb-a746-d8a9fb548b48" path="/var/lib/kubelet/pods/775cb414-3b30-4adb-a746-d8a9fb548b48/volumes" Oct 30 13:30:25.514816 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:25.514802 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5592ca1-ccf3-4297-bc42-ea5aa0cca97a" path="/var/lib/kubelet/pods/a5592ca1-ccf3-4297-bc42-ea5aa0cca97a/volumes" Oct 30 13:30:28.095056 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095018 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr"] Oct 30 13:30:28.095674 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095657 4931 cpu_manager.go:414] "containerMap: removing container" podUID="f23416bb-8d3b-422e-9880-fca313dfd70c" containerName="cleanup" Oct 30 13:30:28.095674 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095674 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f23416bb-8d3b-422e-9880-fca313dfd70c" containerName="cleanup" Oct 30 13:30:28.095801 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095689 4931 cpu_manager.go:414] "containerMap: removing container" podUID="07b99b5b-b3b4-4621-9382-4db0db4bf317" containerName="cleanup" Oct 30 13:30:28.095801 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095696 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07b99b5b-b3b4-4621-9382-4db0db4bf317" containerName="cleanup" Oct 30 13:30:28.095801 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095704 4931 cpu_manager.go:414] "containerMap: removing container" podUID="a3d54ed7-7658-49d1-95e4-2e522546e8b3" containerName="cleanup" Oct 30 13:30:28.095801 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095709 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3d54ed7-7658-49d1-95e4-2e522546e8b3" containerName="cleanup" Oct 30 13:30:28.095801 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095717 4931 cpu_manager.go:414] "containerMap: removing container" podUID="ae4551d9-3f2a-4278-b567-243121f8e1ff" containerName="cleanup" Oct 30 13:30:28.095801 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095722 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae4551d9-3f2a-4278-b567-243121f8e1ff" containerName="cleanup" Oct 30 13:30:28.095801 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095729 4931 cpu_manager.go:414] "containerMap: removing container" podUID="f23416bb-8d3b-422e-9880-fca313dfd70c" containerName="cleanup" Oct 30 13:30:28.095801 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095733 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f23416bb-8d3b-422e-9880-fca313dfd70c" containerName="cleanup" Oct 30 13:30:28.095801 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095745 4931 cpu_manager.go:414] "containerMap: removing container" podUID="32712e88-00fd-4f97-882a-a633da6a16df" containerName="cleanup" Oct 30 13:30:28.095801 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095750 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32712e88-00fd-4f97-882a-a633da6a16df" containerName="cleanup" Oct 30 13:30:28.095801 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095762 4931 cpu_manager.go:414] "containerMap: removing container" podUID="a3d54ed7-7658-49d1-95e4-2e522546e8b3" containerName="cleanup" Oct 30 13:30:28.095801 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095770 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3d54ed7-7658-49d1-95e4-2e522546e8b3" containerName="cleanup" Oct 30 13:30:28.095801 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095784 4931 cpu_manager.go:414] "containerMap: removing container" podUID="775cb414-3b30-4adb-a746-d8a9fb548b48" containerName="cleanup" Oct 30 13:30:28.095801 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095792 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="775cb414-3b30-4adb-a746-d8a9fb548b48" containerName="cleanup" Oct 30 13:30:28.095801 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095803 4931 cpu_manager.go:414] "containerMap: removing container" podUID="775cb414-3b30-4adb-a746-d8a9fb548b48" containerName="cleanup" Oct 30 13:30:28.095801 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095810 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="775cb414-3b30-4adb-a746-d8a9fb548b48" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095822 4931 cpu_manager.go:414] "containerMap: removing container" podUID="32712e88-00fd-4f97-882a-a633da6a16df" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095829 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32712e88-00fd-4f97-882a-a633da6a16df" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095842 4931 cpu_manager.go:414] "containerMap: removing container" podUID="122219fe-8078-42d9-b4ed-7c0333b9013c" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095849 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="122219fe-8078-42d9-b4ed-7c0333b9013c" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095862 4931 cpu_manager.go:414] "containerMap: removing container" podUID="32712e88-00fd-4f97-882a-a633da6a16df" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095869 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32712e88-00fd-4f97-882a-a633da6a16df" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095879 4931 cpu_manager.go:414] "containerMap: removing container" podUID="52d65cd3-640f-46e7-90e9-534c3c990930" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095886 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52d65cd3-640f-46e7-90e9-534c3c990930" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095906 4931 cpu_manager.go:414] "containerMap: removing container" podUID="122219fe-8078-42d9-b4ed-7c0333b9013c" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095913 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="122219fe-8078-42d9-b4ed-7c0333b9013c" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095926 4931 cpu_manager.go:414] "containerMap: removing container" podUID="65f88ff2-f7d4-4207-b2d1-a49a6f052a2c" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095932 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65f88ff2-f7d4-4207-b2d1-a49a6f052a2c" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095942 4931 cpu_manager.go:414] "containerMap: removing container" podUID="a5592ca1-ccf3-4297-bc42-ea5aa0cca97a" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095950 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5592ca1-ccf3-4297-bc42-ea5aa0cca97a" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095961 4931 cpu_manager.go:414] "containerMap: removing container" podUID="a3d54ed7-7658-49d1-95e4-2e522546e8b3" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095968 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3d54ed7-7658-49d1-95e4-2e522546e8b3" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095980 4931 cpu_manager.go:414] "containerMap: removing container" podUID="52d65cd3-640f-46e7-90e9-534c3c990930" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095986 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52d65cd3-640f-46e7-90e9-534c3c990930" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.095998 4931 cpu_manager.go:414] "containerMap: removing container" podUID="ae4551d9-3f2a-4278-b567-243121f8e1ff" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096005 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae4551d9-3f2a-4278-b567-243121f8e1ff" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096017 4931 cpu_manager.go:414] "containerMap: removing container" podUID="07b99b5b-b3b4-4621-9382-4db0db4bf317" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096024 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07b99b5b-b3b4-4621-9382-4db0db4bf317" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096043 4931 cpu_manager.go:414] "containerMap: removing container" podUID="a5592ca1-ccf3-4297-bc42-ea5aa0cca97a" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096051 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5592ca1-ccf3-4297-bc42-ea5aa0cca97a" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096064 4931 cpu_manager.go:414] "containerMap: removing container" podUID="65f88ff2-f7d4-4207-b2d1-a49a6f052a2c" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096071 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65f88ff2-f7d4-4207-b2d1-a49a6f052a2c" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096247 4931 cpu_manager.go:414] "containerMap: removing container" podUID="65f88ff2-f7d4-4207-b2d1-a49a6f052a2c" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096256 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65f88ff2-f7d4-4207-b2d1-a49a6f052a2c" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096283 4931 cpu_manager.go:414] "containerMap: removing container" podUID="a5592ca1-ccf3-4297-bc42-ea5aa0cca97a" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096291 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5592ca1-ccf3-4297-bc42-ea5aa0cca97a" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096505 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f23416bb-8d3b-422e-9880-fca313dfd70c" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096521 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="07b99b5b-b3b4-4621-9382-4db0db4bf317" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096531 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3d54ed7-7658-49d1-95e4-2e522546e8b3" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096541 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae4551d9-3f2a-4278-b567-243121f8e1ff" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096550 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f23416bb-8d3b-422e-9880-fca313dfd70c" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096564 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="32712e88-00fd-4f97-882a-a633da6a16df" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096574 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="07b99b5b-b3b4-4621-9382-4db0db4bf317" containerName="cleanup" Oct 30 13:30:28.096555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096581 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3d54ed7-7658-49d1-95e4-2e522546e8b3" containerName="cleanup" Oct 30 13:30:28.097711 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096590 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="775cb414-3b30-4adb-a746-d8a9fb548b48" containerName="cleanup" Oct 30 13:30:28.097711 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096599 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="775cb414-3b30-4adb-a746-d8a9fb548b48" containerName="cleanup" Oct 30 13:30:28.097711 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096610 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="32712e88-00fd-4f97-882a-a633da6a16df" containerName="cleanup" Oct 30 13:30:28.097711 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096614 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae4551d9-3f2a-4278-b567-243121f8e1ff" containerName="cleanup" Oct 30 13:30:28.097711 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096621 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="122219fe-8078-42d9-b4ed-7c0333b9013c" containerName="cleanup" Oct 30 13:30:28.097711 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096627 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="32712e88-00fd-4f97-882a-a633da6a16df" containerName="cleanup" Oct 30 13:30:28.097711 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096632 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="52d65cd3-640f-46e7-90e9-534c3c990930" containerName="cleanup" Oct 30 13:30:28.097711 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096639 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="122219fe-8078-42d9-b4ed-7c0333b9013c" containerName="cleanup" Oct 30 13:30:28.097711 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096645 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="65f88ff2-f7d4-4207-b2d1-a49a6f052a2c" containerName="cleanup" Oct 30 13:30:28.097711 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096650 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5592ca1-ccf3-4297-bc42-ea5aa0cca97a" containerName="cleanup" Oct 30 13:30:28.097711 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096655 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3d54ed7-7658-49d1-95e4-2e522546e8b3" containerName="cleanup" Oct 30 13:30:28.097711 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096661 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="52d65cd3-640f-46e7-90e9-534c3c990930" containerName="cleanup" Oct 30 13:30:28.097711 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096669 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae4551d9-3f2a-4278-b567-243121f8e1ff" containerName="cleanup" Oct 30 13:30:28.097711 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096676 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="07b99b5b-b3b4-4621-9382-4db0db4bf317" containerName="cleanup" Oct 30 13:30:28.097711 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096685 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="122219fe-8078-42d9-b4ed-7c0333b9013c" containerName="cleanup" Oct 30 13:30:28.097711 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096692 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="65f88ff2-f7d4-4207-b2d1-a49a6f052a2c" containerName="cleanup" Oct 30 13:30:28.097711 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.096697 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5592ca1-ccf3-4297-bc42-ea5aa0cca97a" containerName="cleanup" Oct 30 13:30:28.101599 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.101580 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" Oct 30 13:30:28.124659 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.124632 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-51288b-openshift-default-dockercfg-bbqtx\"" Oct 30 13:30:28.144803 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.144774 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr"] Oct 30 13:30:28.264137 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.264099 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24214a0a-42ce-4fa2-b4ef-9a28771278ce-istio-data\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" Oct 30 13:30:28.264385 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.264147 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24214a0a-42ce-4fa2-b4ef-9a28771278ce-workload-certs\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" Oct 30 13:30:28.264385 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.264351 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24214a0a-42ce-4fa2-b4ef-9a28771278ce-istiod-ca-cert\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" Oct 30 13:30:28.264539 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.264459 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdc9h\" (UniqueName: \"kubernetes.io/projected/24214a0a-42ce-4fa2-b4ef-9a28771278ce-kube-api-access-tdc9h\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" Oct 30 13:30:28.264539 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.264525 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24214a0a-42ce-4fa2-b4ef-9a28771278ce-istio-token\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" Oct 30 13:30:28.264651 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.264582 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24214a0a-42ce-4fa2-b4ef-9a28771278ce-istio-podinfo\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" Oct 30 13:30:28.264651 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.264608 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24214a0a-42ce-4fa2-b4ef-9a28771278ce-credential-socket\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" Oct 30 13:30:28.264762 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.264745 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24214a0a-42ce-4fa2-b4ef-9a28771278ce-workload-socket\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" Oct 30 13:30:28.264885 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.264863 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24214a0a-42ce-4fa2-b4ef-9a28771278ce-istio-envoy\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" Oct 30 13:30:28.366638 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.366558 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24214a0a-42ce-4fa2-b4ef-9a28771278ce-istiod-ca-cert\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" Oct 30 13:30:28.366638 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.366622 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdc9h\" (UniqueName: \"kubernetes.io/projected/24214a0a-42ce-4fa2-b4ef-9a28771278ce-kube-api-access-tdc9h\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" Oct 30 13:30:28.366887 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.366653 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24214a0a-42ce-4fa2-b4ef-9a28771278ce-istio-token\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" Oct 30 13:30:28.366887 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.366671 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24214a0a-42ce-4fa2-b4ef-9a28771278ce-istio-podinfo\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" Oct 30 13:30:28.366887 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.366688 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24214a0a-42ce-4fa2-b4ef-9a28771278ce-credential-socket\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" Oct 30 13:30:28.366887 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.366762 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24214a0a-42ce-4fa2-b4ef-9a28771278ce-workload-socket\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" Oct 30 13:30:28.366887 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.366862 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24214a0a-42ce-4fa2-b4ef-9a28771278ce-istio-envoy\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" Oct 30 13:30:28.367240 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.366894 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24214a0a-42ce-4fa2-b4ef-9a28771278ce-istio-data\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" Oct 30 13:30:28.367240 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.366933 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24214a0a-42ce-4fa2-b4ef-9a28771278ce-workload-certs\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" Oct 30 13:30:28.367240 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.367151 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24214a0a-42ce-4fa2-b4ef-9a28771278ce-credential-socket\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" Oct 30 13:30:28.367240 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.367214 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24214a0a-42ce-4fa2-b4ef-9a28771278ce-workload-socket\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" Oct 30 13:30:28.367500 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.367474 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24214a0a-42ce-4fa2-b4ef-9a28771278ce-istio-data\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" Oct 30 13:30:28.367547 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.367501 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24214a0a-42ce-4fa2-b4ef-9a28771278ce-workload-certs\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" Oct 30 13:30:28.367547 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.367518 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24214a0a-42ce-4fa2-b4ef-9a28771278ce-istiod-ca-cert\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" Oct 30 13:30:28.369764 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.369740 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24214a0a-42ce-4fa2-b4ef-9a28771278ce-istio-envoy\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" Oct 30 13:30:28.370027 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.370005 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24214a0a-42ce-4fa2-b4ef-9a28771278ce-istio-podinfo\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" Oct 30 13:30:28.391549 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.391525 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24214a0a-42ce-4fa2-b4ef-9a28771278ce-istio-token\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" Oct 30 13:30:28.391767 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.391743 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdc9h\" (UniqueName: \"kubernetes.io/projected/24214a0a-42ce-4fa2-b4ef-9a28771278ce-kube-api-access-tdc9h\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" Oct 30 13:30:28.417519 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.417493 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" Oct 30 13:30:28.743438 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:30:28.743400 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24214a0a_42ce_4fa2_b4ef_9a28771278ce.slice/crio-c44b921de9f6301cf69137c24f2eeee69e78e6b29fdf73358fc558c7e0777cfb WatchSource:0}: Error finding container c44b921de9f6301cf69137c24f2eeee69e78e6b29fdf73358fc558c7e0777cfb: Status 404 returned error can't find the container with id c44b921de9f6301cf69137c24f2eeee69e78e6b29fdf73358fc558c7e0777cfb Oct 30 13:30:28.743971 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.743906 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr"] Oct 30 13:30:28.746299 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.746235 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:30:28.746409 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.746310 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:30:28.746409 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:28.746341 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:30:29.002643 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:29.002552 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" event={"ID":"24214a0a-42ce-4fa2-b4ef-9a28771278ce","Type":"ContainerStarted","Data":"96f1aa904a20156b00cbd3a7794a9a350fe787d6bc80c53a90a677ca350f649a"} Oct 30 13:30:29.002643 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:29.002615 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" event={"ID":"24214a0a-42ce-4fa2-b4ef-9a28771278ce","Type":"ContainerStarted","Data":"c44b921de9f6301cf69137c24f2eeee69e78e6b29fdf73358fc558c7e0777cfb"} Oct 30 13:30:29.053025 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:29.052973 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" podStartSLOduration=1.052960662 podStartE2EDuration="1.052960662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:29.050332572 +0000 UTC m=+2727.653831037" watchObservedRunningTime="2026-10-30 13:30:29.052960662 +0000 UTC m=+2727.656459163" Oct 30 13:30:29.418395 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:29.418310 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" Oct 30 13:30:29.419776 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:29.419752 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 13:30:29.419832 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:29.419803 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" podUID="24214a0a-42ce-4fa2-b4ef-9a28771278ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 13:30:30.425288 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:30.420736 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 13:30:30.425288 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:30.420801 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" podUID="24214a0a-42ce-4fa2-b4ef-9a28771278ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 13:30:31.418846 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:31.418807 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 13:30:31.419045 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:31.418862 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" podUID="24214a0a-42ce-4fa2-b4ef-9a28771278ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 13:30:32.418098 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:32.418060 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 13:30:32.418565 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:32.418127 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" podUID="24214a0a-42ce-4fa2-b4ef-9a28771278ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 13:30:33.418188 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:33.418147 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 13:30:33.418640 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:33.418221 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" podUID="24214a0a-42ce-4fa2-b4ef-9a28771278ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 13:30:34.418694 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:34.418657 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 13:30:34.419081 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:34.418729 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" podUID="24214a0a-42ce-4fa2-b4ef-9a28771278ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 13:30:35.424058 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:35.424027 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" Oct 30 13:30:35.424580 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:35.424523 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" Oct 30 13:30:35.425304 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:35.425286 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" Oct 30 13:30:37.361216 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:30:37.361178 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:30:42.584829 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:42.583471 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:30:43.103739 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:43.103703 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84c737ea-7dbkv"] Oct 30 13:30:43.111531 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:43.111500 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84c737ea-7dbkv"] Oct 30 13:30:43.514638 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:43.514607 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ba7ef30-326a-48dd-b501-940c52c0c380" path="/var/lib/kubelet/pods/9ba7ef30-326a-48dd-b501-940c52c0c380/volumes" Oct 30 13:30:48.440812 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.440764 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp"] Oct 30 13:30:48.441435 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.441391 4931 cpu_manager.go:414] "containerMap: removing container" podUID="775cb414-3b30-4adb-a746-d8a9fb548b48" containerName="cleanup" Oct 30 13:30:48.441435 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.441405 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="775cb414-3b30-4adb-a746-d8a9fb548b48" containerName="cleanup" Oct 30 13:30:48.441435 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.441429 4931 cpu_manager.go:414] "containerMap: removing container" podUID="122219fe-8078-42d9-b4ed-7c0333b9013c" containerName="cleanup" Oct 30 13:30:48.441435 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.441437 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="122219fe-8078-42d9-b4ed-7c0333b9013c" containerName="cleanup" Oct 30 13:30:48.441673 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.441449 4931 cpu_manager.go:414] "containerMap: removing container" podUID="52d65cd3-640f-46e7-90e9-534c3c990930" containerName="cleanup" Oct 30 13:30:48.441673 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.441454 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52d65cd3-640f-46e7-90e9-534c3c990930" containerName="cleanup" Oct 30 13:30:48.441673 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.441481 4931 cpu_manager.go:414] "containerMap: removing container" podUID="07b99b5b-b3b4-4621-9382-4db0db4bf317" containerName="cleanup" Oct 30 13:30:48.441673 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.441486 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07b99b5b-b3b4-4621-9382-4db0db4bf317" containerName="cleanup" Oct 30 13:30:48.441673 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.441495 4931 cpu_manager.go:414] "containerMap: removing container" podUID="ae4551d9-3f2a-4278-b567-243121f8e1ff" containerName="cleanup" Oct 30 13:30:48.441673 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.441499 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae4551d9-3f2a-4278-b567-243121f8e1ff" containerName="cleanup" Oct 30 13:30:48.441673 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.441513 4931 cpu_manager.go:414] "containerMap: removing container" podUID="f23416bb-8d3b-422e-9880-fca313dfd70c" containerName="cleanup" Oct 30 13:30:48.441673 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.441520 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f23416bb-8d3b-422e-9880-fca313dfd70c" containerName="cleanup" Oct 30 13:30:48.441673 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.441615 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="52d65cd3-640f-46e7-90e9-534c3c990930" containerName="cleanup" Oct 30 13:30:48.441673 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.441628 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="65f88ff2-f7d4-4207-b2d1-a49a6f052a2c" containerName="cleanup" Oct 30 13:30:48.441673 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.441634 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5592ca1-ccf3-4297-bc42-ea5aa0cca97a" containerName="cleanup" Oct 30 13:30:48.441673 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.441644 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f23416bb-8d3b-422e-9880-fca313dfd70c" containerName="cleanup" Oct 30 13:30:48.441673 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.441652 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="775cb414-3b30-4adb-a746-d8a9fb548b48" containerName="cleanup" Oct 30 13:30:48.446699 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.446675 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp" Oct 30 13:30:48.462808 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.462783 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-a4f385-serving-cert\"" Oct 30 13:30:48.473141 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.473111 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp"] Oct 30 13:30:48.567054 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.567014 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp\" (UID: \"44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp" Oct 30 13:30:48.567211 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.567188 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhktc\" (UniqueName: \"kubernetes.io/projected/44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b-kube-api-access-xhktc\") pod \"maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp\" (UID: \"44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp" Oct 30 13:30:48.567382 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.567364 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp\" (UID: \"44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp" Oct 30 13:30:48.567454 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.567394 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b-metrics-certs\") pod \"maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp\" (UID: \"44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp" Oct 30 13:30:48.635958 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.635924 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adf49f94-tqqq8"] Oct 30 13:30:48.640223 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.640199 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adf49f94-tqqq8" Oct 30 13:30:48.663388 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.663359 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:30:48.663555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.663432 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:30:48.664685 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.664657 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adf49f94-tqqq8"] Oct 30 13:30:48.668370 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.668340 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp\" (UID: \"44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp" Oct 30 13:30:48.668481 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.668387 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b-metrics-certs\") pod \"maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp\" (UID: \"44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp" Oct 30 13:30:48.668585 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.668566 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp\" (UID: \"44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp" Oct 30 13:30:48.668707 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.668690 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhktc\" (UniqueName: \"kubernetes.io/projected/44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b-kube-api-access-xhktc\") pod \"maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp\" (UID: \"44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp" Oct 30 13:30:48.669894 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.669856 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp\" (UID: \"44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp" Oct 30 13:30:48.672614 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.672583 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b-metrics-certs\") pod \"maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp\" (UID: \"44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp" Oct 30 13:30:48.677816 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.677785 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp\" (UID: \"44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp" Oct 30 13:30:48.763618 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.763529 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhktc\" (UniqueName: \"kubernetes.io/projected/44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b-kube-api-access-xhktc\") pod \"maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp\" (UID: \"44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp" Oct 30 13:30:48.771303 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.770802 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1464a40b-fa02-47d9-8f74-3061dedf2390-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adf49f94-tqqq8\" (UID: \"1464a40b-fa02-47d9-8f74-3061dedf2390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adf49f94-tqqq8" Oct 30 13:30:48.771303 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.771024 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1464a40b-fa02-47d9-8f74-3061dedf2390-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adf49f94-tqqq8\" (UID: \"1464a40b-fa02-47d9-8f74-3061dedf2390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adf49f94-tqqq8" Oct 30 13:30:48.771303 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.771142 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stzfl\" (UniqueName: \"kubernetes.io/projected/1464a40b-fa02-47d9-8f74-3061dedf2390-kube-api-access-stzfl\") pod \"maas-api-revoke-sub-adf49f94-tqqq8\" (UID: \"1464a40b-fa02-47d9-8f74-3061dedf2390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adf49f94-tqqq8" Oct 30 13:30:48.872629 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.872571 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1464a40b-fa02-47d9-8f74-3061dedf2390-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adf49f94-tqqq8\" (UID: \"1464a40b-fa02-47d9-8f74-3061dedf2390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adf49f94-tqqq8" Oct 30 13:30:48.872817 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.872707 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stzfl\" (UniqueName: \"kubernetes.io/projected/1464a40b-fa02-47d9-8f74-3061dedf2390-kube-api-access-stzfl\") pod \"maas-api-revoke-sub-adf49f94-tqqq8\" (UID: \"1464a40b-fa02-47d9-8f74-3061dedf2390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adf49f94-tqqq8" Oct 30 13:30:48.872894 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.872815 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1464a40b-fa02-47d9-8f74-3061dedf2390-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adf49f94-tqqq8\" (UID: \"1464a40b-fa02-47d9-8f74-3061dedf2390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adf49f94-tqqq8" Oct 30 13:30:48.873343 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.873319 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1464a40b-fa02-47d9-8f74-3061dedf2390-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adf49f94-tqqq8\" (UID: \"1464a40b-fa02-47d9-8f74-3061dedf2390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adf49f94-tqqq8" Oct 30 13:30:48.875255 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.875230 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1464a40b-fa02-47d9-8f74-3061dedf2390-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adf49f94-tqqq8\" (UID: \"1464a40b-fa02-47d9-8f74-3061dedf2390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adf49f94-tqqq8" Oct 30 13:30:48.897792 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.897762 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stzfl\" (UniqueName: \"kubernetes.io/projected/1464a40b-fa02-47d9-8f74-3061dedf2390-kube-api-access-stzfl\") pod \"maas-api-revoke-sub-adf49f94-tqqq8\" (UID: \"1464a40b-fa02-47d9-8f74-3061dedf2390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adf49f94-tqqq8" Oct 30 13:30:48.954121 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:48.954088 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adf49f94-tqqq8" Oct 30 13:30:49.076247 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:49.075941 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp" Oct 30 13:30:49.316086 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:30:49.316017 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1464a40b_fa02_47d9_8f74_3061dedf2390.slice/crio-13ebfcb30354743a2f5f5f8a817072412250fbddfd64d3c5dc0f393f81ba6943 WatchSource:0}: Error finding container 13ebfcb30354743a2f5f5f8a817072412250fbddfd64d3c5dc0f393f81ba6943: Status 404 returned error can't find the container with id 13ebfcb30354743a2f5f5f8a817072412250fbddfd64d3c5dc0f393f81ba6943 Oct 30 13:30:49.317087 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:49.317058 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adf49f94-tqqq8"] Oct 30 13:30:49.396852 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:49.396820 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adf49f94-tqqq8" event={"ID":"1464a40b-fa02-47d9-8f74-3061dedf2390","Type":"ContainerStarted","Data":"13ebfcb30354743a2f5f5f8a817072412250fbddfd64d3c5dc0f393f81ba6943"} Oct 30 13:30:49.398489 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:30:49.398461 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b6c7ea_cef8_448d_b6ee_5b8ca9063b3b.slice/crio-61f3e420138ecec977ccf36cbc0d50fda0cc775a0855923a697e83b1e9816ee9 WatchSource:0}: Error finding container 61f3e420138ecec977ccf36cbc0d50fda0cc775a0855923a697e83b1e9816ee9: Status 404 returned error can't find the container with id 61f3e420138ecec977ccf36cbc0d50fda0cc775a0855923a697e83b1e9816ee9 Oct 30 13:30:49.400352 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:49.400334 4931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:30:49.400668 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:49.400645 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp"] Oct 30 13:30:50.408449 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:50.408404 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp" event={"ID":"44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b","Type":"ContainerStarted","Data":"1bf3296060610f9f5a422dc7215731e3ffdf6ffb1dd84570e15015713a359bf1"} Oct 30 13:30:50.408995 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:50.408513 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp" event={"ID":"44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b","Type":"ContainerStarted","Data":"61f3e420138ecec977ccf36cbc0d50fda0cc775a0855923a697e83b1e9816ee9"} Oct 30 13:30:50.408995 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:50.408646 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp" Oct 30 13:30:50.410422 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:50.410393 4931 generic.go:356] "Generic (PLEG): container finished" podID="1464a40b-fa02-47d9-8f74-3061dedf2390" containerID="aa45c69f3d282adbd98668437806ccc2ca8f0ab73436decf3cc427e2161a3332" exitCode=0 Oct 30 13:30:50.410578 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:50.410468 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adf49f94-tqqq8" event={"ID":"1464a40b-fa02-47d9-8f74-3061dedf2390","Type":"ContainerDied","Data":"aa45c69f3d282adbd98668437806ccc2ca8f0ab73436decf3cc427e2161a3332"} Oct 30 13:30:50.456119 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:50.456048 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp" podStartSLOduration=2.092362599 podStartE2EDuration="2.45603032s" totalImagesPullingTime="363.667721ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:50.452690183 +0000 UTC m=+2749.056188628" watchObservedRunningTime="2026-10-30 13:30:50.45603032 +0000 UTC m=+2749.059528768" Oct 30 13:30:51.582249 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:51.582219 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adf49f94-tqqq8" Oct 30 13:30:51.718441 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:51.718406 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1464a40b-fa02-47d9-8f74-3061dedf2390-kube-api-access-stzfl\" (UniqueName: \"kubernetes.io/projected/1464a40b-fa02-47d9-8f74-3061dedf2390-kube-api-access-stzfl\") pod \"1464a40b-fa02-47d9-8f74-3061dedf2390\" (UID: \"1464a40b-fa02-47d9-8f74-3061dedf2390\") " Oct 30 13:30:51.718639 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:51.718496 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1464a40b-fa02-47d9-8f74-3061dedf2390-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1464a40b-fa02-47d9-8f74-3061dedf2390-maas-api-xks-ca\") pod \"1464a40b-fa02-47d9-8f74-3061dedf2390\" (UID: \"1464a40b-fa02-47d9-8f74-3061dedf2390\") " Oct 30 13:30:51.718639 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:51.718570 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1464a40b-fa02-47d9-8f74-3061dedf2390-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1464a40b-fa02-47d9-8f74-3061dedf2390-maas-api-openshift-ca\") pod \"1464a40b-fa02-47d9-8f74-3061dedf2390\" (UID: \"1464a40b-fa02-47d9-8f74-3061dedf2390\") " Oct 30 13:30:51.718987 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:51.718952 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1464a40b-fa02-47d9-8f74-3061dedf2390-maas-api-openshift-ca" pod "1464a40b-fa02-47d9-8f74-3061dedf2390" (UID: "1464a40b-fa02-47d9-8f74-3061dedf2390"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:51.719453 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:51.719432 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1464a40b-fa02-47d9-8f74-3061dedf2390-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:30:51.720611 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:51.720588 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1464a40b-fa02-47d9-8f74-3061dedf2390-maas-api-xks-ca" pod "1464a40b-fa02-47d9-8f74-3061dedf2390" (UID: "1464a40b-fa02-47d9-8f74-3061dedf2390"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:30:51.720721 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:51.720707 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1464a40b-fa02-47d9-8f74-3061dedf2390-kube-api-access-stzfl" pod "1464a40b-fa02-47d9-8f74-3061dedf2390" (UID: "1464a40b-fa02-47d9-8f74-3061dedf2390"). InnerVolumeSpecName "kube-api-access-stzfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:51.820325 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:51.820250 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stzfl\" (UniqueName: \"kubernetes.io/projected/1464a40b-fa02-47d9-8f74-3061dedf2390-kube-api-access-stzfl\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:30:51.820325 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:51.820321 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1464a40b-fa02-47d9-8f74-3061dedf2390-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:30:52.443179 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:52.443141 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adf49f94-tqqq8" event={"ID":"1464a40b-fa02-47d9-8f74-3061dedf2390","Type":"ContainerDied","Data":"13ebfcb30354743a2f5f5f8a817072412250fbddfd64d3c5dc0f393f81ba6943"} Oct 30 13:30:52.443416 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:52.443209 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13ebfcb30354743a2f5f5f8a817072412250fbddfd64d3c5dc0f393f81ba6943" Oct 30 13:30:52.443416 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:52.443153 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adf49f94-tqqq8" Oct 30 13:30:54.527003 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:54.526972 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr"] Oct 30 13:30:54.527803 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:54.527779 4931 cpu_manager.go:414] "containerMap: removing container" podUID="1464a40b-fa02-47d9-8f74-3061dedf2390" containerName="revoke-keys" Oct 30 13:30:54.527908 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:54.527804 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1464a40b-fa02-47d9-8f74-3061dedf2390" containerName="revoke-keys" Oct 30 13:30:54.528014 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:54.527999 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="1464a40b-fa02-47d9-8f74-3061dedf2390" containerName="revoke-keys" Oct 30 13:30:54.531795 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:54.531772 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr" Oct 30 13:30:54.557938 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:54.557911 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-03b5b3-serving-cert\"" Oct 30 13:30:54.564617 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:54.564585 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr"] Oct 30 13:30:54.672331 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:54.672233 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr\" (UID: \"2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr" Oct 30 13:30:54.672509 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:54.672432 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr\" (UID: \"2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr" Oct 30 13:30:54.672593 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:54.672574 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr\" (UID: \"2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr" Oct 30 13:30:54.672635 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:54.672607 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7wsk\" (UniqueName: \"kubernetes.io/projected/2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc-kube-api-access-x7wsk\") pod \"maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr\" (UID: \"2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr" Oct 30 13:30:54.782669 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:54.782568 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr\" (UID: \"2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr" Oct 30 13:30:54.782669 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:54.782631 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7wsk\" (UniqueName: \"kubernetes.io/projected/2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc-kube-api-access-x7wsk\") pod \"maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr\" (UID: \"2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr" Oct 30 13:30:54.782924 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:54.782815 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr\" (UID: \"2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr" Oct 30 13:30:54.782924 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:54.782912 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr\" (UID: \"2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr" Oct 30 13:30:54.785078 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:54.784522 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr\" (UID: \"2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr" Oct 30 13:30:54.796287 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:54.792860 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr\" (UID: \"2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr" Oct 30 13:30:54.822126 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:54.819146 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr\" (UID: \"2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr" Oct 30 13:30:54.826320 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:54.826253 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7wsk\" (UniqueName: \"kubernetes.io/projected/2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc-kube-api-access-x7wsk\") pod \"maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr\" (UID: \"2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr" Oct 30 13:30:54.847571 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:54.847540 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr" Oct 30 13:30:55.159321 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:30:55.159289 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e5ba5bf_3b97_4afa_bc8c_09f151c6dfbc.slice/crio-0d6722645bc392e8ed4e38be22c45e58c2f2bcf3ba9ee299c4902d97ac7154e3 WatchSource:0}: Error finding container 0d6722645bc392e8ed4e38be22c45e58c2f2bcf3ba9ee299c4902d97ac7154e3: Status 404 returned error can't find the container with id 0d6722645bc392e8ed4e38be22c45e58c2f2bcf3ba9ee299c4902d97ac7154e3 Oct 30 13:30:55.159943 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:55.159921 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr"] Oct 30 13:30:55.496292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:55.494152 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr" event={"ID":"2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc","Type":"ContainerStarted","Data":"0d6722645bc392e8ed4e38be22c45e58c2f2bcf3ba9ee299c4902d97ac7154e3"} Oct 30 13:30:56.424233 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:56.424203 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp" Oct 30 13:30:56.506777 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:56.506724 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr" event={"ID":"2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc","Type":"ContainerStarted","Data":"3a37f056b790f90747f3a268edcd83510ca4ade4b5f6dcb8c31ed52a378c8a8b"} Oct 30 13:30:56.507081 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:56.507041 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr" Oct 30 13:30:56.554133 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:56.554080 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr" podStartSLOduration=2.187680267 podStartE2EDuration="2.55406633s" totalImagesPullingTime="366.386063ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:56.551639084 +0000 UTC m=+2755.155137581" watchObservedRunningTime="2026-10-30 13:30:56.55406633 +0000 UTC m=+2755.157564835" Oct 30 13:30:59.041753 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:59.041709 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3653b5e-kzxvr"] Oct 30 13:30:59.046414 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:59.046390 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3653b5e-kzxvr" Oct 30 13:30:59.062764 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:59.062714 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:30:59.063004 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:59.062981 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:30:59.066923 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:59.066899 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3653b5e-kzxvr"] Oct 30 13:30:59.179426 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:59.179396 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b64b32-aa7c-4448-bb79-3e93cdfc528a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3653b5e-kzxvr\" (UID: \"b9b64b32-aa7c-4448-bb79-3e93cdfc528a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3653b5e-kzxvr" Oct 30 13:30:59.179588 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:59.179518 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52n25\" (UniqueName: \"kubernetes.io/projected/b9b64b32-aa7c-4448-bb79-3e93cdfc528a-kube-api-access-52n25\") pod \"maas-api-revoke-sub-d3653b5e-kzxvr\" (UID: \"b9b64b32-aa7c-4448-bb79-3e93cdfc528a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3653b5e-kzxvr" Oct 30 13:30:59.179704 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:59.179689 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b64b32-aa7c-4448-bb79-3e93cdfc528a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3653b5e-kzxvr\" (UID: \"b9b64b32-aa7c-4448-bb79-3e93cdfc528a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3653b5e-kzxvr" Oct 30 13:30:59.281382 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:59.281351 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b64b32-aa7c-4448-bb79-3e93cdfc528a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3653b5e-kzxvr\" (UID: \"b9b64b32-aa7c-4448-bb79-3e93cdfc528a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3653b5e-kzxvr" Oct 30 13:30:59.281563 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:59.281444 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b64b32-aa7c-4448-bb79-3e93cdfc528a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3653b5e-kzxvr\" (UID: \"b9b64b32-aa7c-4448-bb79-3e93cdfc528a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3653b5e-kzxvr" Oct 30 13:30:59.281563 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:59.281501 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52n25\" (UniqueName: \"kubernetes.io/projected/b9b64b32-aa7c-4448-bb79-3e93cdfc528a-kube-api-access-52n25\") pod \"maas-api-revoke-sub-d3653b5e-kzxvr\" (UID: \"b9b64b32-aa7c-4448-bb79-3e93cdfc528a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3653b5e-kzxvr" Oct 30 13:30:59.282132 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:59.282109 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b64b32-aa7c-4448-bb79-3e93cdfc528a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3653b5e-kzxvr\" (UID: \"b9b64b32-aa7c-4448-bb79-3e93cdfc528a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3653b5e-kzxvr" Oct 30 13:30:59.283905 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:59.283879 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b64b32-aa7c-4448-bb79-3e93cdfc528a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3653b5e-kzxvr\" (UID: \"b9b64b32-aa7c-4448-bb79-3e93cdfc528a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3653b5e-kzxvr" Oct 30 13:30:59.310092 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:59.310028 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52n25\" (UniqueName: \"kubernetes.io/projected/b9b64b32-aa7c-4448-bb79-3e93cdfc528a-kube-api-access-52n25\") pod \"maas-api-revoke-sub-d3653b5e-kzxvr\" (UID: \"b9b64b32-aa7c-4448-bb79-3e93cdfc528a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3653b5e-kzxvr" Oct 30 13:30:59.361010 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:59.360982 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3653b5e-kzxvr" Oct 30 13:30:59.680201 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:30:59.680165 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9b64b32_aa7c_4448_bb79_3e93cdfc528a.slice/crio-8292e4626e2fbaef50b6ef3af5acd5a06bd51cbb79a30aeb9c018dff256558e1 WatchSource:0}: Error finding container 8292e4626e2fbaef50b6ef3af5acd5a06bd51cbb79a30aeb9c018dff256558e1: Status 404 returned error can't find the container with id 8292e4626e2fbaef50b6ef3af5acd5a06bd51cbb79a30aeb9c018dff256558e1 Oct 30 13:30:59.680807 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:30:59.680738 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3653b5e-kzxvr"] Oct 30 13:31:00.543689 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:00.543650 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3653b5e-kzxvr" event={"ID":"b9b64b32-aa7c-4448-bb79-3e93cdfc528a","Type":"ContainerStarted","Data":"e0729d33f298ee1898d98c0a37160429212f1a32c7580d0abeefa582187a0fd5"} Oct 30 13:31:00.544092 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:00.543703 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3653b5e-kzxvr" event={"ID":"b9b64b32-aa7c-4448-bb79-3e93cdfc528a","Type":"ContainerStarted","Data":"8292e4626e2fbaef50b6ef3af5acd5a06bd51cbb79a30aeb9c018dff256558e1"} Oct 30 13:31:00.590404 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:00.590334 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3653b5e-kzxvr" podStartSLOduration=1.590319472 podStartE2EDuration="1.590319472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:00.587620785 +0000 UTC m=+2759.191119233" watchObservedRunningTime="2026-10-30 13:31:00.590319472 +0000 UTC m=+2759.193817912" Oct 30 13:31:01.556280 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:01.556232 4931 generic.go:356] "Generic (PLEG): container finished" podID="b9b64b32-aa7c-4448-bb79-3e93cdfc528a" containerID="e0729d33f298ee1898d98c0a37160429212f1a32c7580d0abeefa582187a0fd5" exitCode=0 Oct 30 13:31:01.556670 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:01.556302 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3653b5e-kzxvr" event={"ID":"b9b64b32-aa7c-4448-bb79-3e93cdfc528a","Type":"ContainerDied","Data":"e0729d33f298ee1898d98c0a37160429212f1a32c7580d0abeefa582187a0fd5"} Oct 30 13:31:02.521895 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:02.521853 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr" Oct 30 13:31:02.773716 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:02.773657 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3653b5e-kzxvr" Oct 30 13:31:02.833357 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:02.833324 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9b64b32-aa7c-4448-bb79-3e93cdfc528a-kube-api-access-52n25\" (UniqueName: \"kubernetes.io/projected/b9b64b32-aa7c-4448-bb79-3e93cdfc528a-kube-api-access-52n25\") pod \"b9b64b32-aa7c-4448-bb79-3e93cdfc528a\" (UID: \"b9b64b32-aa7c-4448-bb79-3e93cdfc528a\") " Oct 30 13:31:02.833549 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:02.833468 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9b64b32-aa7c-4448-bb79-3e93cdfc528a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b64b32-aa7c-4448-bb79-3e93cdfc528a-maas-api-openshift-ca\") pod \"b9b64b32-aa7c-4448-bb79-3e93cdfc528a\" (UID: \"b9b64b32-aa7c-4448-bb79-3e93cdfc528a\") " Oct 30 13:31:02.833643 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:02.833627 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9b64b32-aa7c-4448-bb79-3e93cdfc528a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b64b32-aa7c-4448-bb79-3e93cdfc528a-maas-api-xks-ca\") pod \"b9b64b32-aa7c-4448-bb79-3e93cdfc528a\" (UID: \"b9b64b32-aa7c-4448-bb79-3e93cdfc528a\") " Oct 30 13:31:02.833851 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:02.833819 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9b64b32-aa7c-4448-bb79-3e93cdfc528a-maas-api-openshift-ca" pod "b9b64b32-aa7c-4448-bb79-3e93cdfc528a" (UID: "b9b64b32-aa7c-4448-bb79-3e93cdfc528a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:31:02.834280 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:02.834246 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b64b32-aa7c-4448-bb79-3e93cdfc528a-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:31:02.842125 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:02.840406 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9b64b32-aa7c-4448-bb79-3e93cdfc528a-maas-api-xks-ca" pod "b9b64b32-aa7c-4448-bb79-3e93cdfc528a" (UID: "b9b64b32-aa7c-4448-bb79-3e93cdfc528a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:31:02.842453 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:02.842423 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9b64b32-aa7c-4448-bb79-3e93cdfc528a-kube-api-access-52n25" pod "b9b64b32-aa7c-4448-bb79-3e93cdfc528a" (UID: "b9b64b32-aa7c-4448-bb79-3e93cdfc528a"). InnerVolumeSpecName "kube-api-access-52n25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:31:02.955615 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:02.955572 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52n25\" (UniqueName: \"kubernetes.io/projected/b9b64b32-aa7c-4448-bb79-3e93cdfc528a-kube-api-access-52n25\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:31:02.955878 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:02.955856 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b64b32-aa7c-4448-bb79-3e93cdfc528a-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:31:03.572796 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:03.572764 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3653b5e-kzxvr" Oct 30 13:31:03.572929 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:03.572810 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3653b5e-kzxvr" event={"ID":"b9b64b32-aa7c-4448-bb79-3e93cdfc528a","Type":"ContainerDied","Data":"8292e4626e2fbaef50b6ef3af5acd5a06bd51cbb79a30aeb9c018dff256558e1"} Oct 30 13:31:03.572929 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:03.572852 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8292e4626e2fbaef50b6ef3af5acd5a06bd51cbb79a30aeb9c018dff256558e1" Oct 30 13:31:07.353909 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:31:07.353857 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:31:37.353820 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:31:37.353782 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:31:39.478420 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:39.478388 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj"] Oct 30 13:31:39.479016 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:39.479002 4931 cpu_manager.go:414] "containerMap: removing container" podUID="b9b64b32-aa7c-4448-bb79-3e93cdfc528a" containerName="revoke-keys" Oct 30 13:31:39.479061 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:39.479017 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9b64b32-aa7c-4448-bb79-3e93cdfc528a" containerName="revoke-keys" Oct 30 13:31:39.479152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:39.479142 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9b64b32-aa7c-4448-bb79-3e93cdfc528a" containerName="revoke-keys" Oct 30 13:31:39.483156 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:39.483137 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj" Oct 30 13:31:39.503983 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:39.503960 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-51288b\"/\"openshift-service-ca.crt\"" Oct 30 13:31:39.504097 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:39.503979 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-51288b\"/\"default-dockercfg-vqgb5\"" Oct 30 13:31:39.523490 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:39.523461 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj"] Oct 30 13:31:39.568754 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:39.568714 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-51288b\"/\"rate-test-model-51288b-kserve-self-signed-certs\"" Oct 30 13:31:39.604617 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:39.604586 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-51288b\"/\"kube-root-ca.crt\"" Oct 30 13:31:39.608634 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:39.608595 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm2wr\" (UniqueName: \"kubernetes.io/projected/6972ffa9-c402-415f-8166-7db0e5d8097c-kube-api-access-jm2wr\") pod \"rate-test-model-51288b-kserve-5b466c777c-knhfj\" (UID: \"6972ffa9-c402-415f-8166-7db0e5d8097c\") " pod="ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj" Oct 30 13:31:39.608794 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:39.608769 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6972ffa9-c402-415f-8166-7db0e5d8097c-model-cache\") pod \"rate-test-model-51288b-kserve-5b466c777c-knhfj\" (UID: \"6972ffa9-c402-415f-8166-7db0e5d8097c\") " pod="ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj" Oct 30 13:31:39.608934 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:39.608911 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6972ffa9-c402-415f-8166-7db0e5d8097c-dshm\") pod \"rate-test-model-51288b-kserve-5b466c777c-knhfj\" (UID: \"6972ffa9-c402-415f-8166-7db0e5d8097c\") " pod="ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj" Oct 30 13:31:39.609024 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:39.609003 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6972ffa9-c402-415f-8166-7db0e5d8097c-home\") pod \"rate-test-model-51288b-kserve-5b466c777c-knhfj\" (UID: \"6972ffa9-c402-415f-8166-7db0e5d8097c\") " pod="ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj" Oct 30 13:31:39.609203 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:39.609178 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6972ffa9-c402-415f-8166-7db0e5d8097c-tmp-dir\") pod \"rate-test-model-51288b-kserve-5b466c777c-knhfj\" (UID: \"6972ffa9-c402-415f-8166-7db0e5d8097c\") " pod="ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj" Oct 30 13:31:39.609349 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:39.609219 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6972ffa9-c402-415f-8166-7db0e5d8097c-tls-certs\") pod \"rate-test-model-51288b-kserve-5b466c777c-knhfj\" (UID: \"6972ffa9-c402-415f-8166-7db0e5d8097c\") " pod="ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj" Oct 30 13:31:39.714292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:39.712196 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6972ffa9-c402-415f-8166-7db0e5d8097c-tmp-dir\") pod \"rate-test-model-51288b-kserve-5b466c777c-knhfj\" (UID: \"6972ffa9-c402-415f-8166-7db0e5d8097c\") " pod="ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj" Oct 30 13:31:39.718691 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:39.712256 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6972ffa9-c402-415f-8166-7db0e5d8097c-tls-certs\") pod \"rate-test-model-51288b-kserve-5b466c777c-knhfj\" (UID: \"6972ffa9-c402-415f-8166-7db0e5d8097c\") " pod="ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj" Oct 30 13:31:39.718691 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:39.716722 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm2wr\" (UniqueName: \"kubernetes.io/projected/6972ffa9-c402-415f-8166-7db0e5d8097c-kube-api-access-jm2wr\") pod \"rate-test-model-51288b-kserve-5b466c777c-knhfj\" (UID: \"6972ffa9-c402-415f-8166-7db0e5d8097c\") " pod="ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj" Oct 30 13:31:39.718691 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:39.716837 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6972ffa9-c402-415f-8166-7db0e5d8097c-model-cache\") pod \"rate-test-model-51288b-kserve-5b466c777c-knhfj\" (UID: \"6972ffa9-c402-415f-8166-7db0e5d8097c\") " pod="ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj" Oct 30 13:31:39.718691 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:39.716976 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6972ffa9-c402-415f-8166-7db0e5d8097c-dshm\") pod \"rate-test-model-51288b-kserve-5b466c777c-knhfj\" (UID: \"6972ffa9-c402-415f-8166-7db0e5d8097c\") " pod="ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj" Oct 30 13:31:39.718691 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:39.717036 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6972ffa9-c402-415f-8166-7db0e5d8097c-home\") pod \"rate-test-model-51288b-kserve-5b466c777c-knhfj\" (UID: \"6972ffa9-c402-415f-8166-7db0e5d8097c\") " pod="ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj" Oct 30 13:31:39.718691 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:39.717778 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6972ffa9-c402-415f-8166-7db0e5d8097c-home\") pod \"rate-test-model-51288b-kserve-5b466c777c-knhfj\" (UID: \"6972ffa9-c402-415f-8166-7db0e5d8097c\") " pod="ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj" Oct 30 13:31:39.718691 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:39.718098 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6972ffa9-c402-415f-8166-7db0e5d8097c-tmp-dir\") pod \"rate-test-model-51288b-kserve-5b466c777c-knhfj\" (UID: \"6972ffa9-c402-415f-8166-7db0e5d8097c\") " pod="ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj" Oct 30 13:31:39.728292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:39.726686 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6972ffa9-c402-415f-8166-7db0e5d8097c-model-cache\") pod \"rate-test-model-51288b-kserve-5b466c777c-knhfj\" (UID: \"6972ffa9-c402-415f-8166-7db0e5d8097c\") " pod="ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj" Oct 30 13:31:39.752341 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:39.748838 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6972ffa9-c402-415f-8166-7db0e5d8097c-dshm\") pod \"rate-test-model-51288b-kserve-5b466c777c-knhfj\" (UID: \"6972ffa9-c402-415f-8166-7db0e5d8097c\") " pod="ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj" Oct 30 13:31:39.752341 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:39.749942 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6972ffa9-c402-415f-8166-7db0e5d8097c-tls-certs\") pod \"rate-test-model-51288b-kserve-5b466c777c-knhfj\" (UID: \"6972ffa9-c402-415f-8166-7db0e5d8097c\") " pod="ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj" Oct 30 13:31:39.773292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:39.758131 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm2wr\" (UniqueName: \"kubernetes.io/projected/6972ffa9-c402-415f-8166-7db0e5d8097c-kube-api-access-jm2wr\") pod \"rate-test-model-51288b-kserve-5b466c777c-knhfj\" (UID: \"6972ffa9-c402-415f-8166-7db0e5d8097c\") " pod="ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj" Oct 30 13:31:39.805303 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:39.798547 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj" Oct 30 13:31:40.201067 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:31:40.201022 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6972ffa9_c402_415f_8166_7db0e5d8097c.slice/crio-62a32e63fbf4fbb549faf84bd0ad6580f98f6123584c64015e7afc7fac569216 WatchSource:0}: Error finding container 62a32e63fbf4fbb549faf84bd0ad6580f98f6123584c64015e7afc7fac569216: Status 404 returned error can't find the container with id 62a32e63fbf4fbb549faf84bd0ad6580f98f6123584c64015e7afc7fac569216 Oct 30 13:31:40.203150 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:40.203117 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj"] Oct 30 13:31:41.126304 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:41.115648 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj" event={"ID":"6972ffa9-c402-415f-8166-7db0e5d8097c","Type":"ContainerStarted","Data":"7a0ea744c9a0e49ae5ed367056c75215c405e9d57ff28b51f108a6b32e5bcb43"} Oct 30 13:31:41.126304 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:41.115719 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj" event={"ID":"6972ffa9-c402-415f-8166-7db0e5d8097c","Type":"ContainerStarted","Data":"62a32e63fbf4fbb549faf84bd0ad6580f98f6123584c64015e7afc7fac569216"} Oct 30 13:31:41.200290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:41.199437 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj" podStartSLOduration=2.199415283 podStartE2EDuration="2.199415283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:41.182898314 +0000 UTC m=+2799.786396761" watchObservedRunningTime="2026-10-30 13:31:41.199415283 +0000 UTC m=+2799.802913732" Oct 30 13:31:41.442242 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:41.442207 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng"] Oct 30 13:31:41.449530 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:41.449494 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng" Oct 30 13:31:41.466536 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:41.466509 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng"] Oct 30 13:31:41.470725 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:41.470692 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-76405c\"" Oct 30 13:31:41.474219 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:41.474191 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-76405c-dockercfg-xpqnk\"" Oct 30 13:31:41.491371 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:41.491333 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bf6m\" (UniqueName: \"kubernetes.io/projected/20802417-843e-455e-a55f-b0972b386ac6-kube-api-access-7bf6m\") pod \"payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng\" (UID: \"20802417-843e-455e-a55f-b0972b386ac6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng" Oct 30 13:31:41.491516 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:41.491393 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20802417-843e-455e-a55f-b0972b386ac6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng\" (UID: \"20802417-843e-455e-a55f-b0972b386ac6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng" Oct 30 13:31:41.592893 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:41.592853 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bf6m\" (UniqueName: \"kubernetes.io/projected/20802417-843e-455e-a55f-b0972b386ac6-kube-api-access-7bf6m\") pod \"payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng\" (UID: \"20802417-843e-455e-a55f-b0972b386ac6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng" Oct 30 13:31:41.593103 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:41.592911 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20802417-843e-455e-a55f-b0972b386ac6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng\" (UID: \"20802417-843e-455e-a55f-b0972b386ac6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng" Oct 30 13:31:41.593806 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:41.593781 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20802417-843e-455e-a55f-b0972b386ac6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng\" (UID: \"20802417-843e-455e-a55f-b0972b386ac6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng" Oct 30 13:31:41.617881 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:41.617845 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bf6m\" (UniqueName: \"kubernetes.io/projected/20802417-843e-455e-a55f-b0972b386ac6-kube-api-access-7bf6m\") pod \"payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng\" (UID: \"20802417-843e-455e-a55f-b0972b386ac6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng" Oct 30 13:31:41.768546 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:41.768446 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng" Oct 30 13:31:41.799327 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:41.799296 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj" Oct 30 13:31:42.108155 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:42.106869 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng"] Oct 30 13:31:42.125765 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:42.125728 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng" event={"ID":"20802417-843e-455e-a55f-b0972b386ac6","Type":"ContainerStarted","Data":"3307156ccb940260ac91408b679f7023399cfc29b84693ae640b339a2ad21c92"} Oct 30 13:31:44.152136 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:44.152096 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng" event={"ID":"20802417-843e-455e-a55f-b0972b386ac6","Type":"ContainerStarted","Data":"380e66aa97d55fe278e930828429ff5ceb2a371fe5f0b27130756c3de1059cb9"} Oct 30 13:31:44.153183 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:44.153157 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng" Oct 30 13:31:44.202180 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:44.202126 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng" podStartSLOduration=1.719310836 podStartE2EDuration="3.202111452s" totalImagesPullingTime="1.482800616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:44.196942239 +0000 UTC m=+2802.800440686" watchObservedRunningTime="2026-10-30 13:31:44.202111452 +0000 UTC m=+2802.805609948" Oct 30 13:31:49.800038 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:49.800003 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj" Oct 30 13:31:49.819187 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:49.819160 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj" Oct 30 13:31:50.290258 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:50.290225 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj" Oct 30 13:31:52.108909 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:52.108872 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74fc974f-9fcx4"] Oct 30 13:31:52.119313 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:52.119236 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74fc974f-9fcx4"] Oct 30 13:31:52.616599 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:52.616564 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp"] Oct 30 13:31:52.622553 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:52.622526 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp" Oct 30 13:31:52.646605 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:52.646578 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp"] Oct 30 13:31:52.654629 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:52.654604 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1ba901\"/\"kube-root-ca.crt\"" Oct 30 13:31:52.654961 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:52.654912 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1ba901\"/\"openshift-service-ca.crt\"" Oct 30 13:31:52.655794 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:52.655772 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1ba901\"/\"auth-test-model-1ba901-kserve-self-signed-certs\"" Oct 30 13:31:52.656019 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:52.655998 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1ba901\"/\"default-dockercfg-g5h5g\"" Oct 30 13:31:52.820776 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:52.820735 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5601047-51b4-4310-9289-580b8891746e-tls-certs\") pod \"auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp\" (UID: \"c5601047-51b4-4310-9289-580b8891746e\") " pod="ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp" Oct 30 13:31:52.820985 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:52.820896 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2zhv\" (UniqueName: \"kubernetes.io/projected/c5601047-51b4-4310-9289-580b8891746e-kube-api-access-k2zhv\") pod \"auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp\" (UID: \"c5601047-51b4-4310-9289-580b8891746e\") " pod="ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp" Oct 30 13:31:52.820985 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:52.820943 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5601047-51b4-4310-9289-580b8891746e-home\") pod \"auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp\" (UID: \"c5601047-51b4-4310-9289-580b8891746e\") " pod="ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp" Oct 30 13:31:52.821129 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:52.821051 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5601047-51b4-4310-9289-580b8891746e-model-cache\") pod \"auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp\" (UID: \"c5601047-51b4-4310-9289-580b8891746e\") " pod="ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp" Oct 30 13:31:52.821194 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:52.821178 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5601047-51b4-4310-9289-580b8891746e-dshm\") pod \"auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp\" (UID: \"c5601047-51b4-4310-9289-580b8891746e\") " pod="ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp" Oct 30 13:31:52.821253 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:52.821223 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5601047-51b4-4310-9289-580b8891746e-tmp-dir\") pod \"auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp\" (UID: \"c5601047-51b4-4310-9289-580b8891746e\") " pod="ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp" Oct 30 13:31:52.925971 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:52.925814 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5601047-51b4-4310-9289-580b8891746e-tls-certs\") pod \"auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp\" (UID: \"c5601047-51b4-4310-9289-580b8891746e\") " pod="ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp" Oct 30 13:31:52.927230 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:52.927198 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2zhv\" (UniqueName: \"kubernetes.io/projected/c5601047-51b4-4310-9289-580b8891746e-kube-api-access-k2zhv\") pod \"auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp\" (UID: \"c5601047-51b4-4310-9289-580b8891746e\") " pod="ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp" Oct 30 13:31:52.927534 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:52.927511 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5601047-51b4-4310-9289-580b8891746e-home\") pod \"auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp\" (UID: \"c5601047-51b4-4310-9289-580b8891746e\") " pod="ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp" Oct 30 13:31:52.927712 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:52.927696 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5601047-51b4-4310-9289-580b8891746e-model-cache\") pod \"auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp\" (UID: \"c5601047-51b4-4310-9289-580b8891746e\") " pod="ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp" Oct 30 13:31:52.927913 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:52.927897 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5601047-51b4-4310-9289-580b8891746e-dshm\") pod \"auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp\" (UID: \"c5601047-51b4-4310-9289-580b8891746e\") " pod="ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp" Oct 30 13:31:52.928047 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:52.928032 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5601047-51b4-4310-9289-580b8891746e-tmp-dir\") pod \"auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp\" (UID: \"c5601047-51b4-4310-9289-580b8891746e\") " pod="ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp" Oct 30 13:31:52.943883 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:52.928605 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5601047-51b4-4310-9289-580b8891746e-home\") pod \"auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp\" (UID: \"c5601047-51b4-4310-9289-580b8891746e\") " pod="ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp" Oct 30 13:31:52.944837 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:52.944807 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5601047-51b4-4310-9289-580b8891746e-tmp-dir\") pod \"auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp\" (UID: \"c5601047-51b4-4310-9289-580b8891746e\") " pod="ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp" Oct 30 13:31:52.959908 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:52.946861 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5601047-51b4-4310-9289-580b8891746e-model-cache\") pod \"auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp\" (UID: \"c5601047-51b4-4310-9289-580b8891746e\") " pod="ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp" Oct 30 13:31:52.959908 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:52.959859 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2zhv\" (UniqueName: \"kubernetes.io/projected/c5601047-51b4-4310-9289-580b8891746e-kube-api-access-k2zhv\") pod \"auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp\" (UID: \"c5601047-51b4-4310-9289-580b8891746e\") " pod="ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp" Oct 30 13:31:52.976657 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:52.976625 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5601047-51b4-4310-9289-580b8891746e-tls-certs\") pod \"auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp\" (UID: \"c5601047-51b4-4310-9289-580b8891746e\") " pod="ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp" Oct 30 13:31:52.979567 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:52.979511 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5601047-51b4-4310-9289-580b8891746e-dshm\") pod \"auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp\" (UID: \"c5601047-51b4-4310-9289-580b8891746e\") " pod="ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp" Oct 30 13:31:53.238958 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:53.238920 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp" Oct 30 13:31:53.515025 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:53.514950 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cbbc99e-cb4e-4388-a71f-fa4e87d6555e" path="/var/lib/kubelet/pods/5cbbc99e-cb4e-4388-a71f-fa4e87d6555e/volumes" Oct 30 13:31:53.555643 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:53.555614 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp"] Oct 30 13:31:53.555643 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:31:53.555625 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5601047_51b4_4310_9289_580b8891746e.slice/crio-ea490111ceadc59c5412a40fd9e29d1216c75095f3b4e724969d7fdd81accbfd WatchSource:0}: Error finding container ea490111ceadc59c5412a40fd9e29d1216c75095f3b4e724969d7fdd81accbfd: Status 404 returned error can't find the container with id ea490111ceadc59c5412a40fd9e29d1216c75095f3b4e724969d7fdd81accbfd Oct 30 13:31:54.281328 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:54.281284 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp" event={"ID":"c5601047-51b4-4310-9289-580b8891746e","Type":"ContainerStarted","Data":"2699bf11018dbdbd981a905373cba75d3029c0b22511e87e414137719688c5c4"} Oct 30 13:31:54.281892 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:54.281343 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp" event={"ID":"c5601047-51b4-4310-9289-580b8891746e","Type":"ContainerStarted","Data":"ea490111ceadc59c5412a40fd9e29d1216c75095f3b4e724969d7fdd81accbfd"} Oct 30 13:31:54.329616 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:54.329556 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp" podStartSLOduration=2.329540812 podStartE2EDuration="2.329540812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:54.326746603 +0000 UTC m=+2812.930245051" watchObservedRunningTime="2026-10-30 13:31:54.329540812 +0000 UTC m=+2812.933039306" Oct 30 13:31:54.781438 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:54.781391 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:31:55.239124 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:55.239081 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp" Oct 30 13:31:56.198108 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:31:56.198066 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng" Oct 30 13:32:03.239921 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:03.239882 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp" Oct 30 13:32:03.257891 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:03.257852 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp" Oct 30 13:32:03.417913 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:03.417880 4931 scope.go:122] "RemoveContainer" containerID="1acd0a1cdf72e17c2ec19236ae0fbbe463209ff423b810abe1cf19569a582e11" Oct 30 13:32:03.429659 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:03.429385 4931 scope.go:122] "RemoveContainer" containerID="be65020a256f468edba8d916c5a57b71b02ac388176ab33b84992c4e0346b2ee" Oct 30 13:32:03.439764 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:03.439741 4931 scope.go:122] "RemoveContainer" containerID="34728b82ad869405b788a8202216648475e62adda843252179e01cacdcce1d30" Oct 30 13:32:03.633547 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:03.633464 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp" Oct 30 13:32:07.063468 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:07.063424 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a71c333e-9bxjf"] Oct 30 13:32:07.080086 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:07.080056 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a71c333e-9bxjf"] Oct 30 13:32:07.432867 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:32:07.432754 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:32:07.514025 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:07.513993 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ef3c3b3-766a-4eda-9734-819f35478218" path="/var/lib/kubelet/pods/4ef3c3b3-766a-4eda-9734-819f35478218/volumes" Oct 30 13:32:11.377974 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:11.377938 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7"] Oct 30 13:32:11.383922 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:11.383899 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" Oct 30 13:32:11.402431 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:11.402409 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-3c1b28-openshift-default-dockercfg-rbqkl\"" Oct 30 13:32:11.420658 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:11.420631 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7"] Oct 30 13:32:11.446812 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:11.446783 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9babe7d-51a7-4d00-ab49-76441dd6dec5-istio-podinfo\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" Oct 30 13:32:11.446957 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:11.446840 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9babe7d-51a7-4d00-ab49-76441dd6dec5-workload-certs\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" Oct 30 13:32:11.446957 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:11.446865 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9babe7d-51a7-4d00-ab49-76441dd6dec5-istio-data\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" Oct 30 13:32:11.446957 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:11.446927 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9babe7d-51a7-4d00-ab49-76441dd6dec5-istiod-ca-cert\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" Oct 30 13:32:11.447167 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:11.447096 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4jgj\" (UniqueName: \"kubernetes.io/projected/a9babe7d-51a7-4d00-ab49-76441dd6dec5-kube-api-access-w4jgj\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" Oct 30 13:32:11.447167 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:11.447144 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9babe7d-51a7-4d00-ab49-76441dd6dec5-credential-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" Oct 30 13:32:11.447303 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:11.447227 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9babe7d-51a7-4d00-ab49-76441dd6dec5-workload-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" Oct 30 13:32:11.447377 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:11.447305 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9babe7d-51a7-4d00-ab49-76441dd6dec5-istio-envoy\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" Oct 30 13:32:11.447377 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:11.447330 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9babe7d-51a7-4d00-ab49-76441dd6dec5-istio-token\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" Oct 30 13:32:11.548851 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:11.548815 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9babe7d-51a7-4d00-ab49-76441dd6dec5-istio-podinfo\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" Oct 30 13:32:11.549060 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:11.548872 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9babe7d-51a7-4d00-ab49-76441dd6dec5-workload-certs\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" Oct 30 13:32:11.549060 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:11.548898 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9babe7d-51a7-4d00-ab49-76441dd6dec5-istio-data\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" Oct 30 13:32:11.549060 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:11.548930 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9babe7d-51a7-4d00-ab49-76441dd6dec5-istiod-ca-cert\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" Oct 30 13:32:11.549060 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:11.549030 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4jgj\" (UniqueName: \"kubernetes.io/projected/a9babe7d-51a7-4d00-ab49-76441dd6dec5-kube-api-access-w4jgj\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" Oct 30 13:32:11.549060 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:11.549056 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9babe7d-51a7-4d00-ab49-76441dd6dec5-credential-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" Oct 30 13:32:11.549449 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:11.549093 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9babe7d-51a7-4d00-ab49-76441dd6dec5-workload-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" Oct 30 13:32:11.549449 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:11.549135 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9babe7d-51a7-4d00-ab49-76441dd6dec5-istio-envoy\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" Oct 30 13:32:11.549449 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:11.549165 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9babe7d-51a7-4d00-ab49-76441dd6dec5-istio-token\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" Oct 30 13:32:11.549626 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:11.549440 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9babe7d-51a7-4d00-ab49-76441dd6dec5-workload-certs\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" Oct 30 13:32:11.549716 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:11.549693 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9babe7d-51a7-4d00-ab49-76441dd6dec5-credential-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" Oct 30 13:32:11.549990 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:11.549958 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9babe7d-51a7-4d00-ab49-76441dd6dec5-istio-data\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" Oct 30 13:32:11.550517 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:11.550493 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9babe7d-51a7-4d00-ab49-76441dd6dec5-workload-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" Oct 30 13:32:11.550641 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:11.550620 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9babe7d-51a7-4d00-ab49-76441dd6dec5-istiod-ca-cert\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" Oct 30 13:32:11.552445 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:11.552419 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9babe7d-51a7-4d00-ab49-76441dd6dec5-istio-envoy\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" Oct 30 13:32:11.553124 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:11.553098 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9babe7d-51a7-4d00-ab49-76441dd6dec5-istio-podinfo\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" Oct 30 13:32:11.581858 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:11.581828 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9babe7d-51a7-4d00-ab49-76441dd6dec5-istio-token\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" Oct 30 13:32:11.586517 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:11.586489 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4jgj\" (UniqueName: \"kubernetes.io/projected/a9babe7d-51a7-4d00-ab49-76441dd6dec5-kube-api-access-w4jgj\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" Oct 30 13:32:11.700928 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:11.700551 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" Oct 30 13:32:12.052806 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:32:12.052772 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9babe7d_51a7_4d00_ab49_76441dd6dec5.slice/crio-3919e3ee761378c8d2520974003323826c0085a2e44dd44c392ca0b29e44e818 WatchSource:0}: Error finding container 3919e3ee761378c8d2520974003323826c0085a2e44dd44c392ca0b29e44e818: Status 404 returned error can't find the container with id 3919e3ee761378c8d2520974003323826c0085a2e44dd44c392ca0b29e44e818 Oct 30 13:32:12.054042 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:12.054018 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7"] Oct 30 13:32:12.055886 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:12.055856 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:32:12.055944 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:12.055929 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:32:12.055992 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:12.055978 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:32:12.736902 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:12.736863 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" event={"ID":"a9babe7d-51a7-4d00-ab49-76441dd6dec5","Type":"ContainerStarted","Data":"2e3198b0df152359f3a8fb485a11d9f4c9f2d5ecb996eaae31ee1d0b009db496"} Oct 30 13:32:12.737304 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:12.736919 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" event={"ID":"a9babe7d-51a7-4d00-ab49-76441dd6dec5","Type":"ContainerStarted","Data":"3919e3ee761378c8d2520974003323826c0085a2e44dd44c392ca0b29e44e818"} Oct 30 13:32:12.801919 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:12.801861 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" podStartSLOduration=1.801847709 podStartE2EDuration="1.801847709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:12.800765976 +0000 UTC m=+2831.404264434" watchObservedRunningTime="2026-10-30 13:32:12.801847709 +0000 UTC m=+2831.405346154" Oct 30 13:32:13.711221 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:13.711167 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" Oct 30 13:32:13.711781 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:13.711749 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 13:32:13.712056 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:13.711976 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" podUID="a9babe7d-51a7-4d00-ab49-76441dd6dec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 13:32:14.702500 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:14.702459 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 13:32:14.703008 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:14.702526 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" podUID="a9babe7d-51a7-4d00-ab49-76441dd6dec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 13:32:15.702181 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:15.702145 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 13:32:15.702464 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:15.702441 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" podUID="a9babe7d-51a7-4d00-ab49-76441dd6dec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 13:32:16.702203 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:16.702168 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 13:32:16.702634 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:16.702217 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" podUID="a9babe7d-51a7-4d00-ab49-76441dd6dec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 13:32:17.702282 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:17.702238 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 13:32:17.702720 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:17.702322 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" podUID="a9babe7d-51a7-4d00-ab49-76441dd6dec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 13:32:18.669361 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:18.669320 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw"] Oct 30 13:32:18.676332 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:18.676298 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" Oct 30 13:32:18.694805 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:18.694776 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-96ab73-openshift-default-dockercfg-nshzl\"" Oct 30 13:32:18.702234 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:18.702204 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 13:32:18.702427 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:18.702256 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" podUID="a9babe7d-51a7-4d00-ab49-76441dd6dec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 13:32:18.716531 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:18.716488 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw"] Oct 30 13:32:18.780905 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:18.780865 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db26d540-4441-49aa-be9f-51edab84b99c-istio-podinfo\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" Oct 30 13:32:18.781198 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:18.781182 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dkhw\" (UniqueName: \"kubernetes.io/projected/db26d540-4441-49aa-be9f-51edab84b99c-kube-api-access-4dkhw\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" Oct 30 13:32:18.781371 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:18.781338 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db26d540-4441-49aa-be9f-51edab84b99c-istio-token\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" Oct 30 13:32:18.781446 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:18.781429 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db26d540-4441-49aa-be9f-51edab84b99c-istiod-ca-cert\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" Oct 30 13:32:18.781484 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:18.781464 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db26d540-4441-49aa-be9f-51edab84b99c-istio-envoy\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" Oct 30 13:32:18.781524 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:18.781500 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db26d540-4441-49aa-be9f-51edab84b99c-workload-socket\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" Oct 30 13:32:18.781559 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:18.781533 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db26d540-4441-49aa-be9f-51edab84b99c-credential-socket\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" Oct 30 13:32:18.781779 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:18.781758 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db26d540-4441-49aa-be9f-51edab84b99c-istio-data\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" Oct 30 13:32:18.781918 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:18.781905 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db26d540-4441-49aa-be9f-51edab84b99c-workload-certs\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" Oct 30 13:32:18.884221 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:18.884169 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dkhw\" (UniqueName: \"kubernetes.io/projected/db26d540-4441-49aa-be9f-51edab84b99c-kube-api-access-4dkhw\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" Oct 30 13:32:18.884440 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:18.884251 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db26d540-4441-49aa-be9f-51edab84b99c-istio-token\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" Oct 30 13:32:18.884440 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:18.884397 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db26d540-4441-49aa-be9f-51edab84b99c-istiod-ca-cert\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" Oct 30 13:32:18.884579 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:18.884438 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db26d540-4441-49aa-be9f-51edab84b99c-istio-envoy\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" Oct 30 13:32:18.884579 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:18.884486 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db26d540-4441-49aa-be9f-51edab84b99c-workload-socket\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" Oct 30 13:32:18.884579 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:18.884531 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db26d540-4441-49aa-be9f-51edab84b99c-credential-socket\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" Oct 30 13:32:18.884757 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:18.884742 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db26d540-4441-49aa-be9f-51edab84b99c-istio-data\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" Oct 30 13:32:18.884833 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:18.884811 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db26d540-4441-49aa-be9f-51edab84b99c-workload-certs\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" Oct 30 13:32:18.884897 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:18.884866 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db26d540-4441-49aa-be9f-51edab84b99c-workload-socket\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" Oct 30 13:32:18.884897 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:18.884882 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db26d540-4441-49aa-be9f-51edab84b99c-istio-podinfo\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" Oct 30 13:32:18.885035 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:18.885003 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db26d540-4441-49aa-be9f-51edab84b99c-credential-socket\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" Oct 30 13:32:18.885222 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:18.885202 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db26d540-4441-49aa-be9f-51edab84b99c-istiod-ca-cert\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" Oct 30 13:32:18.885338 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:18.885315 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db26d540-4441-49aa-be9f-51edab84b99c-istio-data\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" Oct 30 13:32:18.885411 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:18.885355 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db26d540-4441-49aa-be9f-51edab84b99c-workload-certs\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" Oct 30 13:32:18.886904 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:18.886881 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db26d540-4441-49aa-be9f-51edab84b99c-istio-envoy\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" Oct 30 13:32:18.887520 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:18.887496 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db26d540-4441-49aa-be9f-51edab84b99c-istio-podinfo\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" Oct 30 13:32:18.909224 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:18.909189 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db26d540-4441-49aa-be9f-51edab84b99c-istio-token\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" Oct 30 13:32:18.909561 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:18.909512 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dkhw\" (UniqueName: \"kubernetes.io/projected/db26d540-4441-49aa-be9f-51edab84b99c-kube-api-access-4dkhw\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" Oct 30 13:32:18.999168 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:18.999126 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" Oct 30 13:32:19.349466 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:19.349431 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw"] Oct 30 13:32:19.349756 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:32:19.349730 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb26d540_4441_49aa_be9f_51edab84b99c.slice/crio-0d792b7f239d80c08be48da062f9bf6b0b90c76986f383a65275337fa418593f WatchSource:0}: Error finding container 0d792b7f239d80c08be48da062f9bf6b0b90c76986f383a65275337fa418593f: Status 404 returned error can't find the container with id 0d792b7f239d80c08be48da062f9bf6b0b90c76986f383a65275337fa418593f Oct 30 13:32:19.352612 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:19.352578 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:32:19.352708 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:19.352657 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:32:19.352779 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:19.352703 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:32:19.708438 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:19.708398 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" Oct 30 13:32:19.709094 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:19.709066 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" Oct 30 13:32:19.710475 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:19.710446 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" Oct 30 13:32:19.897842 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:19.897668 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" event={"ID":"db26d540-4441-49aa-be9f-51edab84b99c","Type":"ContainerStarted","Data":"5818472411fbdb6fe85ea3df7c0044ccb9f4f25d61bf7fe97baec779d9a1e93a"} Oct 30 13:32:19.897842 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:19.897794 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" event={"ID":"db26d540-4441-49aa-be9f-51edab84b99c","Type":"ContainerStarted","Data":"0d792b7f239d80c08be48da062f9bf6b0b90c76986f383a65275337fa418593f"} Oct 30 13:32:19.995477 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:19.995358 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" podStartSLOduration=1.995339733 podStartE2EDuration="1.995339733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:19.96123457 +0000 UTC m=+2838.564733042" watchObservedRunningTime="2026-10-30 13:32:19.995339733 +0000 UTC m=+2838.598838181" Oct 30 13:32:20.000634 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:20.000598 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" Oct 30 13:32:20.002130 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:20.002099 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 13:32:20.002376 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:20.002351 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" podUID="db26d540-4441-49aa-be9f-51edab84b99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 13:32:21.000368 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:21.000330 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 13:32:21.000913 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:21.000387 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" podUID="db26d540-4441-49aa-be9f-51edab84b99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 13:32:21.662691 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:21.662637 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7"] Oct 30 13:32:21.683314 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:21.678744 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" Oct 30 13:32:21.690260 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:21.690228 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7"] Oct 30 13:32:21.757574 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:21.757531 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-03b5b3-openshift-default-dockercfg-dp4wq\"" Oct 30 13:32:21.860294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:21.859460 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/550e0c64-4028-4fb3-98ef-fdc52cc66189-istio-data\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" Oct 30 13:32:21.860294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:21.859542 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlprd\" (UniqueName: \"kubernetes.io/projected/550e0c64-4028-4fb3-98ef-fdc52cc66189-kube-api-access-nlprd\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" Oct 30 13:32:21.860294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:21.859607 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/550e0c64-4028-4fb3-98ef-fdc52cc66189-istio-token\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" Oct 30 13:32:21.860294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:21.859681 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/550e0c64-4028-4fb3-98ef-fdc52cc66189-workload-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" Oct 30 13:32:21.860294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:21.859729 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/550e0c64-4028-4fb3-98ef-fdc52cc66189-istiod-ca-cert\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" Oct 30 13:32:21.860294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:21.859771 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/550e0c64-4028-4fb3-98ef-fdc52cc66189-workload-certs\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" Oct 30 13:32:21.860294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:21.859892 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/550e0c64-4028-4fb3-98ef-fdc52cc66189-credential-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" Oct 30 13:32:21.860294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:21.859982 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/550e0c64-4028-4fb3-98ef-fdc52cc66189-istio-podinfo\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" Oct 30 13:32:21.860294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:21.860056 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/550e0c64-4028-4fb3-98ef-fdc52cc66189-istio-envoy\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" Oct 30 13:32:21.961829 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:21.961792 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/550e0c64-4028-4fb3-98ef-fdc52cc66189-workload-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" Oct 30 13:32:21.961995 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:21.961849 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/550e0c64-4028-4fb3-98ef-fdc52cc66189-istiod-ca-cert\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" Oct 30 13:32:21.961995 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:21.961886 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/550e0c64-4028-4fb3-98ef-fdc52cc66189-workload-certs\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" Oct 30 13:32:21.961995 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:21.961960 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/550e0c64-4028-4fb3-98ef-fdc52cc66189-credential-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" Oct 30 13:32:21.962113 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:21.962018 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/550e0c64-4028-4fb3-98ef-fdc52cc66189-istio-podinfo\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" Oct 30 13:32:21.962113 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:21.962068 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/550e0c64-4028-4fb3-98ef-fdc52cc66189-istio-envoy\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" Oct 30 13:32:21.962180 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:21.962125 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/550e0c64-4028-4fb3-98ef-fdc52cc66189-istio-data\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" Oct 30 13:32:21.962180 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:21.962153 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlprd\" (UniqueName: \"kubernetes.io/projected/550e0c64-4028-4fb3-98ef-fdc52cc66189-kube-api-access-nlprd\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" Oct 30 13:32:21.962251 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:21.962195 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/550e0c64-4028-4fb3-98ef-fdc52cc66189-istio-token\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" Oct 30 13:32:21.962919 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:21.962892 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/550e0c64-4028-4fb3-98ef-fdc52cc66189-istio-data\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" Oct 30 13:32:21.963057 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:21.962945 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/550e0c64-4028-4fb3-98ef-fdc52cc66189-workload-certs\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" Oct 30 13:32:21.963057 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:21.963024 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/550e0c64-4028-4fb3-98ef-fdc52cc66189-workload-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" Oct 30 13:32:21.963175 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:21.963076 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/550e0c64-4028-4fb3-98ef-fdc52cc66189-credential-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" Oct 30 13:32:21.963730 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:21.963705 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/550e0c64-4028-4fb3-98ef-fdc52cc66189-istiod-ca-cert\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" Oct 30 13:32:21.964978 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:21.964956 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/550e0c64-4028-4fb3-98ef-fdc52cc66189-istio-envoy\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" Oct 30 13:32:21.966048 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:21.966022 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/550e0c64-4028-4fb3-98ef-fdc52cc66189-istio-podinfo\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" Oct 30 13:32:21.987431 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:21.987398 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlprd\" (UniqueName: \"kubernetes.io/projected/550e0c64-4028-4fb3-98ef-fdc52cc66189-kube-api-access-nlprd\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" Oct 30 13:32:21.987617 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:21.987589 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/550e0c64-4028-4fb3-98ef-fdc52cc66189-istio-token\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" Oct 30 13:32:22.000631 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:22.000599 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 13:32:22.001011 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:22.000660 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" podUID="db26d540-4441-49aa-be9f-51edab84b99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 13:32:22.003845 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:22.003824 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" Oct 30 13:32:22.370985 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:22.370580 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7"] Oct 30 13:32:22.382005 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:22.381946 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:32:22.382363 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:22.382331 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:32:22.382604 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:22.382578 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:32:22.923319 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:22.923274 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" event={"ID":"550e0c64-4028-4fb3-98ef-fdc52cc66189","Type":"ContainerStarted","Data":"8c3e9c17377496135ef0edd2b29975097898c4bb897436fdd62c038eab997056"} Oct 30 13:32:22.923522 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:22.923357 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" event={"ID":"550e0c64-4028-4fb3-98ef-fdc52cc66189","Type":"ContainerStarted","Data":"d202387ecdf26573391309111cf3d5799c08c2a5e0ddf01732ee11ef67096b5c"} Oct 30 13:32:22.994953 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:22.994882 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" podStartSLOduration=1.994867245 podStartE2EDuration="1.994867245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:22.991732006 +0000 UTC m=+2841.595230445" watchObservedRunningTime="2026-10-30 13:32:22.994867245 +0000 UTC m=+2841.598365690" Oct 30 13:32:23.000487 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:23.000447 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 13:32:23.000663 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:23.000519 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" podUID="db26d540-4441-49aa-be9f-51edab84b99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 13:32:23.004136 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:23.004112 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" Oct 30 13:32:23.005320 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:23.005300 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 13:32:23.005411 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:23.005336 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" podUID="550e0c64-4028-4fb3-98ef-fdc52cc66189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 13:32:24.001048 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:24.000965 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 13:32:24.001606 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:24.001249 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" podUID="db26d540-4441-49aa-be9f-51edab84b99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 13:32:24.004092 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:24.004067 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 13:32:24.004213 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:24.004102 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" podUID="550e0c64-4028-4fb3-98ef-fdc52cc66189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 13:32:24.999911 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:24.999871 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 13:32:25.000100 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:24.999930 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" podUID="db26d540-4441-49aa-be9f-51edab84b99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 13:32:25.004239 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:25.004201 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 13:32:25.004687 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:25.004260 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" podUID="550e0c64-4028-4fb3-98ef-fdc52cc66189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 13:32:26.000960 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:26.000901 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 13:32:26.001148 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:26.001001 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" podUID="db26d540-4441-49aa-be9f-51edab84b99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 13:32:26.004334 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:26.004304 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 13:32:26.004688 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:26.004358 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" podUID="550e0c64-4028-4fb3-98ef-fdc52cc66189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 13:32:27.004915 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:27.004876 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 13:32:27.005420 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:27.004948 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" podUID="550e0c64-4028-4fb3-98ef-fdc52cc66189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 13:32:27.005633 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:27.005611 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" Oct 30 13:32:27.006119 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:27.006090 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" Oct 30 13:32:27.006790 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:27.006771 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" Oct 30 13:32:28.004553 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:28.004518 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 13:32:28.004722 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:28.004581 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" podUID="550e0c64-4028-4fb3-98ef-fdc52cc66189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 13:32:28.175096 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:28.175041 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts"] Oct 30 13:32:28.185643 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:28.185615 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" Oct 30 13:32:28.200858 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:28.200823 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts"] Oct 30 13:32:28.206381 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:28.206349 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-627dfa\"/\"openshift-service-ca.crt\"" Oct 30 13:32:28.206564 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:28.206366 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-627dfa\"/\"rate-test-model-627dfa-kserve-self-signed-certs\"" Oct 30 13:32:28.218288 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:28.218247 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-627dfa\"/\"kube-root-ca.crt\"" Oct 30 13:32:28.218513 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:28.218355 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-627dfa\"/\"default-dockercfg-5f5xw\"" Oct 30 13:32:28.310003 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:28.309909 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d31d29-93c6-4992-b3b7-1bb491b382fb-tls-certs\") pod \"rate-test-model-627dfa-kserve-d56cb8b85-4f2ts\" (UID: \"b0d31d29-93c6-4992-b3b7-1bb491b382fb\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" Oct 30 13:32:28.310003 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:28.309989 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csj5c\" (UniqueName: \"kubernetes.io/projected/b0d31d29-93c6-4992-b3b7-1bb491b382fb-kube-api-access-csj5c\") pod \"rate-test-model-627dfa-kserve-d56cb8b85-4f2ts\" (UID: \"b0d31d29-93c6-4992-b3b7-1bb491b382fb\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" Oct 30 13:32:28.310205 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:28.310128 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d31d29-93c6-4992-b3b7-1bb491b382fb-tmp-dir\") pod \"rate-test-model-627dfa-kserve-d56cb8b85-4f2ts\" (UID: \"b0d31d29-93c6-4992-b3b7-1bb491b382fb\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" Oct 30 13:32:28.310245 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:28.310216 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d31d29-93c6-4992-b3b7-1bb491b382fb-home\") pod \"rate-test-model-627dfa-kserve-d56cb8b85-4f2ts\" (UID: \"b0d31d29-93c6-4992-b3b7-1bb491b382fb\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" Oct 30 13:32:28.310245 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:28.310239 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d31d29-93c6-4992-b3b7-1bb491b382fb-dshm\") pod \"rate-test-model-627dfa-kserve-d56cb8b85-4f2ts\" (UID: \"b0d31d29-93c6-4992-b3b7-1bb491b382fb\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" Oct 30 13:32:28.310380 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:28.310362 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d31d29-93c6-4992-b3b7-1bb491b382fb-model-cache\") pod \"rate-test-model-627dfa-kserve-d56cb8b85-4f2ts\" (UID: \"b0d31d29-93c6-4992-b3b7-1bb491b382fb\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" Oct 30 13:32:28.412180 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:28.412134 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d31d29-93c6-4992-b3b7-1bb491b382fb-home\") pod \"rate-test-model-627dfa-kserve-d56cb8b85-4f2ts\" (UID: \"b0d31d29-93c6-4992-b3b7-1bb491b382fb\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" Oct 30 13:32:28.412180 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:28.412176 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d31d29-93c6-4992-b3b7-1bb491b382fb-dshm\") pod \"rate-test-model-627dfa-kserve-d56cb8b85-4f2ts\" (UID: \"b0d31d29-93c6-4992-b3b7-1bb491b382fb\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" Oct 30 13:32:28.412478 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:28.412318 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d31d29-93c6-4992-b3b7-1bb491b382fb-model-cache\") pod \"rate-test-model-627dfa-kserve-d56cb8b85-4f2ts\" (UID: \"b0d31d29-93c6-4992-b3b7-1bb491b382fb\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" Oct 30 13:32:28.412478 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:28.412445 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d31d29-93c6-4992-b3b7-1bb491b382fb-tls-certs\") pod \"rate-test-model-627dfa-kserve-d56cb8b85-4f2ts\" (UID: \"b0d31d29-93c6-4992-b3b7-1bb491b382fb\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" Oct 30 13:32:28.412588 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:28.412505 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csj5c\" (UniqueName: \"kubernetes.io/projected/b0d31d29-93c6-4992-b3b7-1bb491b382fb-kube-api-access-csj5c\") pod \"rate-test-model-627dfa-kserve-d56cb8b85-4f2ts\" (UID: \"b0d31d29-93c6-4992-b3b7-1bb491b382fb\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" Oct 30 13:32:28.412667 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:28.412643 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d31d29-93c6-4992-b3b7-1bb491b382fb-home\") pod \"rate-test-model-627dfa-kserve-d56cb8b85-4f2ts\" (UID: \"b0d31d29-93c6-4992-b3b7-1bb491b382fb\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" Oct 30 13:32:28.412874 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:28.412751 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d31d29-93c6-4992-b3b7-1bb491b382fb-model-cache\") pod \"rate-test-model-627dfa-kserve-d56cb8b85-4f2ts\" (UID: \"b0d31d29-93c6-4992-b3b7-1bb491b382fb\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" Oct 30 13:32:28.412971 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:28.412800 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d31d29-93c6-4992-b3b7-1bb491b382fb-tmp-dir\") pod \"rate-test-model-627dfa-kserve-d56cb8b85-4f2ts\" (UID: \"b0d31d29-93c6-4992-b3b7-1bb491b382fb\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" Oct 30 13:32:28.413087 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:28.413061 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d31d29-93c6-4992-b3b7-1bb491b382fb-tmp-dir\") pod \"rate-test-model-627dfa-kserve-d56cb8b85-4f2ts\" (UID: \"b0d31d29-93c6-4992-b3b7-1bb491b382fb\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" Oct 30 13:32:28.415539 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:28.415509 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d31d29-93c6-4992-b3b7-1bb491b382fb-dshm\") pod \"rate-test-model-627dfa-kserve-d56cb8b85-4f2ts\" (UID: \"b0d31d29-93c6-4992-b3b7-1bb491b382fb\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" Oct 30 13:32:28.415739 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:28.415719 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d31d29-93c6-4992-b3b7-1bb491b382fb-tls-certs\") pod \"rate-test-model-627dfa-kserve-d56cb8b85-4f2ts\" (UID: \"b0d31d29-93c6-4992-b3b7-1bb491b382fb\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" Oct 30 13:32:28.462418 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:28.462371 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csj5c\" (UniqueName: \"kubernetes.io/projected/b0d31d29-93c6-4992-b3b7-1bb491b382fb-kube-api-access-csj5c\") pod \"rate-test-model-627dfa-kserve-d56cb8b85-4f2ts\" (UID: \"b0d31d29-93c6-4992-b3b7-1bb491b382fb\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" Oct 30 13:32:28.507898 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:28.507593 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" Oct 30 13:32:29.005036 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:29.004988 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 13:32:29.005244 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:29.005054 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" podUID="550e0c64-4028-4fb3-98ef-fdc52cc66189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 13:32:29.218080 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:29.218047 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts"] Oct 30 13:32:29.218711 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:32:29.218683 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0d31d29_93c6_4992_b3b7_1bb491b382fb.slice/crio-83da0a8fb4e9d5e15641f6fa4beca7444e2e368d086f40469b895dad352b44d2 WatchSource:0}: Error finding container 83da0a8fb4e9d5e15641f6fa4beca7444e2e368d086f40469b895dad352b44d2: Status 404 returned error can't find the container with id 83da0a8fb4e9d5e15641f6fa4beca7444e2e368d086f40469b895dad352b44d2 Oct 30 13:32:30.019300 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:30.017360 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" Oct 30 13:32:30.019300 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:30.018606 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" Oct 30 13:32:30.019610 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:30.019586 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" Oct 30 13:32:30.021461 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:30.020525 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" event={"ID":"b0d31d29-93c6-4992-b3b7-1bb491b382fb","Type":"ContainerStarted","Data":"5b17843d200f30bc3806479fb4bc0545eae029717d7eec98edba750477787d09"} Oct 30 13:32:30.021461 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:30.020565 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" event={"ID":"b0d31d29-93c6-4992-b3b7-1bb491b382fb","Type":"ContainerStarted","Data":"83da0a8fb4e9d5e15641f6fa4beca7444e2e368d086f40469b895dad352b44d2"} Oct 30 13:32:30.189441 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:30.189388 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" podStartSLOduration=2.1893688510000002 podStartE2EDuration="2.189368851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:30.187890367 +0000 UTC m=+2848.791388832" watchObservedRunningTime="2026-10-30 13:32:30.189368851 +0000 UTC m=+2848.792867299" Oct 30 13:32:30.509208 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:30.509154 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" Oct 30 13:32:37.353603 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:32:37.353560 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:32:38.509255 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:38.509223 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" Oct 30 13:32:38.522766 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:38.522732 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" Oct 30 13:32:39.159632 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:39.159602 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" Oct 30 13:32:42.574154 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:42.569752 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:32:45.219301 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:45.218899 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" podUID="5a504237-e5a3-491f-8f84-0419cf502f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:32:45.290302 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:45.288834 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj" podUID="6972ffa9-c402-415f-8166-7db0e5d8097c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:32:45.388442 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:45.388363 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" podUID="f3b342db-9a01-46ad-b476-bfc912d9ce1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:32:45.425845 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:45.425779 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" podUID="e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 13:32:45.549438 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:45.549334 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" podUID="65799c14-8de4-4b4a-90df-7cd257e1e581" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:32:45.629346 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:45.625542 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp" podUID="c5601047-51b4-4310-9289-580b8891746e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:32:46.975777 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:46.975740 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-558a020d-x9w4g"] Oct 30 13:32:46.981440 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:46.981417 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-558a020d-x9w4g" Oct 30 13:32:47.004677 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:47.004645 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-558a020d-x9w4g"] Oct 30 13:32:47.012317 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:47.011579 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:32:47.012317 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:47.011729 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:32:47.045235 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:47.045199 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22a99a6c-3118-44f4-b80a-cab1053c0b36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-558a020d-x9w4g\" (UID: \"22a99a6c-3118-44f4-b80a-cab1053c0b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-558a020d-x9w4g" Oct 30 13:32:47.045467 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:47.045379 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22a99a6c-3118-44f4-b80a-cab1053c0b36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-558a020d-x9w4g\" (UID: \"22a99a6c-3118-44f4-b80a-cab1053c0b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-558a020d-x9w4g" Oct 30 13:32:47.045622 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:47.045599 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv5xd\" (UniqueName: \"kubernetes.io/projected/22a99a6c-3118-44f4-b80a-cab1053c0b36-kube-api-access-hv5xd\") pod \"maas-api-revoke-sub-558a020d-x9w4g\" (UID: \"22a99a6c-3118-44f4-b80a-cab1053c0b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-558a020d-x9w4g" Oct 30 13:32:47.146733 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:47.146689 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv5xd\" (UniqueName: \"kubernetes.io/projected/22a99a6c-3118-44f4-b80a-cab1053c0b36-kube-api-access-hv5xd\") pod \"maas-api-revoke-sub-558a020d-x9w4g\" (UID: \"22a99a6c-3118-44f4-b80a-cab1053c0b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-558a020d-x9w4g" Oct 30 13:32:47.146939 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:47.146856 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22a99a6c-3118-44f4-b80a-cab1053c0b36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-558a020d-x9w4g\" (UID: \"22a99a6c-3118-44f4-b80a-cab1053c0b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-558a020d-x9w4g" Oct 30 13:32:47.146939 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:47.146934 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22a99a6c-3118-44f4-b80a-cab1053c0b36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-558a020d-x9w4g\" (UID: \"22a99a6c-3118-44f4-b80a-cab1053c0b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-558a020d-x9w4g" Oct 30 13:32:47.147715 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:47.147688 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22a99a6c-3118-44f4-b80a-cab1053c0b36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-558a020d-x9w4g\" (UID: \"22a99a6c-3118-44f4-b80a-cab1053c0b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-558a020d-x9w4g" Oct 30 13:32:47.149388 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:47.149366 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22a99a6c-3118-44f4-b80a-cab1053c0b36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-558a020d-x9w4g\" (UID: \"22a99a6c-3118-44f4-b80a-cab1053c0b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-558a020d-x9w4g" Oct 30 13:32:47.172365 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:47.172335 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv5xd\" (UniqueName: \"kubernetes.io/projected/22a99a6c-3118-44f4-b80a-cab1053c0b36-kube-api-access-hv5xd\") pod \"maas-api-revoke-sub-558a020d-x9w4g\" (UID: \"22a99a6c-3118-44f4-b80a-cab1053c0b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-558a020d-x9w4g" Oct 30 13:32:47.298080 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:47.297997 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-558a020d-x9w4g" Oct 30 13:32:47.615561 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:32:47.615527 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a99a6c_3118_44f4_b80a_cab1053c0b36.slice/crio-2148922216aac990fd51380526d738dc250a19478dba0e184fa3a2674ef22c1a WatchSource:0}: Error finding container 2148922216aac990fd51380526d738dc250a19478dba0e184fa3a2674ef22c1a: Status 404 returned error can't find the container with id 2148922216aac990fd51380526d738dc250a19478dba0e184fa3a2674ef22c1a Oct 30 13:32:47.615751 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:47.615724 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-558a020d-x9w4g"] Oct 30 13:32:48.483485 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:48.483450 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-558a020d-x9w4g" event={"ID":"22a99a6c-3118-44f4-b80a-cab1053c0b36","Type":"ContainerStarted","Data":"5817c5f03afa1158a786dbd430b3b2f9b8269ea4bc5573f6fc18e37ca7df626f"} Oct 30 13:32:48.483935 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:48.483510 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-558a020d-x9w4g" event={"ID":"22a99a6c-3118-44f4-b80a-cab1053c0b36","Type":"ContainerStarted","Data":"2148922216aac990fd51380526d738dc250a19478dba0e184fa3a2674ef22c1a"} Oct 30 13:32:48.531718 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:48.531641 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-558a020d-x9w4g" podStartSLOduration=2.531620388 podStartE2EDuration="2.531620388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:48.525796396 +0000 UTC m=+2867.129294845" watchObservedRunningTime="2026-10-30 13:32:48.531620388 +0000 UTC m=+2867.135118837" Oct 30 13:32:49.499019 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:49.498975 4931 generic.go:356] "Generic (PLEG): container finished" podID="22a99a6c-3118-44f4-b80a-cab1053c0b36" containerID="5817c5f03afa1158a786dbd430b3b2f9b8269ea4bc5573f6fc18e37ca7df626f" exitCode=0 Oct 30 13:32:49.499557 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:49.499243 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-558a020d-x9w4g" event={"ID":"22a99a6c-3118-44f4-b80a-cab1053c0b36","Type":"ContainerDied","Data":"5817c5f03afa1158a786dbd430b3b2f9b8269ea4bc5573f6fc18e37ca7df626f"} Oct 30 13:32:50.676247 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:50.676215 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-558a020d-x9w4g" Oct 30 13:32:50.703866 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:50.703836 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22a99a6c-3118-44f4-b80a-cab1053c0b36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22a99a6c-3118-44f4-b80a-cab1053c0b36-maas-api-xks-ca\") pod \"22a99a6c-3118-44f4-b80a-cab1053c0b36\" (UID: \"22a99a6c-3118-44f4-b80a-cab1053c0b36\") " Oct 30 13:32:50.704033 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:50.704021 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22a99a6c-3118-44f4-b80a-cab1053c0b36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22a99a6c-3118-44f4-b80a-cab1053c0b36-maas-api-openshift-ca\") pod \"22a99a6c-3118-44f4-b80a-cab1053c0b36\" (UID: \"22a99a6c-3118-44f4-b80a-cab1053c0b36\") " Oct 30 13:32:50.704105 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:50.704059 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22a99a6c-3118-44f4-b80a-cab1053c0b36-kube-api-access-hv5xd\" (UniqueName: \"kubernetes.io/projected/22a99a6c-3118-44f4-b80a-cab1053c0b36-kube-api-access-hv5xd\") pod \"22a99a6c-3118-44f4-b80a-cab1053c0b36\" (UID: \"22a99a6c-3118-44f4-b80a-cab1053c0b36\") " Oct 30 13:32:50.704426 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:50.704400 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22a99a6c-3118-44f4-b80a-cab1053c0b36-maas-api-openshift-ca" pod "22a99a6c-3118-44f4-b80a-cab1053c0b36" (UID: "22a99a6c-3118-44f4-b80a-cab1053c0b36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:32:50.704720 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:50.704696 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22a99a6c-3118-44f4-b80a-cab1053c0b36-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:32:50.706047 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:50.706023 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22a99a6c-3118-44f4-b80a-cab1053c0b36-maas-api-xks-ca" pod "22a99a6c-3118-44f4-b80a-cab1053c0b36" (UID: "22a99a6c-3118-44f4-b80a-cab1053c0b36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:32:50.706166 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:50.706142 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22a99a6c-3118-44f4-b80a-cab1053c0b36-kube-api-access-hv5xd" pod "22a99a6c-3118-44f4-b80a-cab1053c0b36" (UID: "22a99a6c-3118-44f4-b80a-cab1053c0b36"). InnerVolumeSpecName "kube-api-access-hv5xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:32:50.806675 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:50.806590 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv5xd\" (UniqueName: \"kubernetes.io/projected/22a99a6c-3118-44f4-b80a-cab1053c0b36-kube-api-access-hv5xd\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:32:50.806675 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:50.806634 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22a99a6c-3118-44f4-b80a-cab1053c0b36-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:32:51.243866 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:51.243828 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7bd69dd-m7lbc"] Oct 30 13:32:51.244611 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:51.244586 4931 cpu_manager.go:414] "containerMap: removing container" podUID="22a99a6c-3118-44f4-b80a-cab1053c0b36" containerName="revoke-keys" Oct 30 13:32:51.244611 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:51.244609 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22a99a6c-3118-44f4-b80a-cab1053c0b36" containerName="revoke-keys" Oct 30 13:32:51.244835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:51.244821 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="22a99a6c-3118-44f4-b80a-cab1053c0b36" containerName="revoke-keys" Oct 30 13:32:51.249050 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:51.249026 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7bd69dd-m7lbc" Oct 30 13:32:51.270714 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:51.270680 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7bd69dd-m7lbc"] Oct 30 13:32:51.314562 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:51.314517 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/676089c8-e20a-4aa2-8220-be872d92bead-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7bd69dd-m7lbc\" (UID: \"676089c8-e20a-4aa2-8220-be872d92bead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7bd69dd-m7lbc" Oct 30 13:32:51.314767 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:51.314633 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/676089c8-e20a-4aa2-8220-be872d92bead-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7bd69dd-m7lbc\" (UID: \"676089c8-e20a-4aa2-8220-be872d92bead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7bd69dd-m7lbc" Oct 30 13:32:51.314767 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:51.314663 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zntq7\" (UniqueName: \"kubernetes.io/projected/676089c8-e20a-4aa2-8220-be872d92bead-kube-api-access-zntq7\") pod \"maas-api-revoke-sub-c7bd69dd-m7lbc\" (UID: \"676089c8-e20a-4aa2-8220-be872d92bead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7bd69dd-m7lbc" Oct 30 13:32:51.415970 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:51.415937 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/676089c8-e20a-4aa2-8220-be872d92bead-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7bd69dd-m7lbc\" (UID: \"676089c8-e20a-4aa2-8220-be872d92bead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7bd69dd-m7lbc" Oct 30 13:32:51.416158 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:51.415976 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zntq7\" (UniqueName: \"kubernetes.io/projected/676089c8-e20a-4aa2-8220-be872d92bead-kube-api-access-zntq7\") pod \"maas-api-revoke-sub-c7bd69dd-m7lbc\" (UID: \"676089c8-e20a-4aa2-8220-be872d92bead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7bd69dd-m7lbc" Oct 30 13:32:51.416357 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:51.416333 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/676089c8-e20a-4aa2-8220-be872d92bead-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7bd69dd-m7lbc\" (UID: \"676089c8-e20a-4aa2-8220-be872d92bead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7bd69dd-m7lbc" Oct 30 13:32:51.416767 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:51.416749 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/676089c8-e20a-4aa2-8220-be872d92bead-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7bd69dd-m7lbc\" (UID: \"676089c8-e20a-4aa2-8220-be872d92bead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7bd69dd-m7lbc" Oct 30 13:32:51.418724 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:51.418703 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/676089c8-e20a-4aa2-8220-be872d92bead-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7bd69dd-m7lbc\" (UID: \"676089c8-e20a-4aa2-8220-be872d92bead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7bd69dd-m7lbc" Oct 30 13:32:51.440835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:51.440801 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zntq7\" (UniqueName: \"kubernetes.io/projected/676089c8-e20a-4aa2-8220-be872d92bead-kube-api-access-zntq7\") pod \"maas-api-revoke-sub-c7bd69dd-m7lbc\" (UID: \"676089c8-e20a-4aa2-8220-be872d92bead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7bd69dd-m7lbc" Oct 30 13:32:51.518779 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:51.518704 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-558a020d-x9w4g" Oct 30 13:32:51.518779 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:51.518721 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-558a020d-x9w4g" event={"ID":"22a99a6c-3118-44f4-b80a-cab1053c0b36","Type":"ContainerDied","Data":"2148922216aac990fd51380526d738dc250a19478dba0e184fa3a2674ef22c1a"} Oct 30 13:32:51.518779 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:51.518777 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2148922216aac990fd51380526d738dc250a19478dba0e184fa3a2674ef22c1a" Oct 30 13:32:51.565510 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:51.565479 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7bd69dd-m7lbc" Oct 30 13:32:51.882301 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:32:51.882220 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod676089c8_e20a_4aa2_8220_be872d92bead.slice/crio-361b4da2bdf77f5be7926c43dcbf2c765aa0cf30dcc150b32cfdc8c701b021ab WatchSource:0}: Error finding container 361b4da2bdf77f5be7926c43dcbf2c765aa0cf30dcc150b32cfdc8c701b021ab: Status 404 returned error can't find the container with id 361b4da2bdf77f5be7926c43dcbf2c765aa0cf30dcc150b32cfdc8c701b021ab Oct 30 13:32:51.883447 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:51.883412 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7bd69dd-m7lbc"] Oct 30 13:32:52.585998 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:52.581711 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7bd69dd-m7lbc" event={"ID":"676089c8-e20a-4aa2-8220-be872d92bead","Type":"ContainerStarted","Data":"20d5cf0f619190ff7dc8ed463717b550ae60a828e9fbc10a37887a50704e27af"} Oct 30 13:32:52.585998 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:52.581788 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7bd69dd-m7lbc" event={"ID":"676089c8-e20a-4aa2-8220-be872d92bead","Type":"ContainerStarted","Data":"361b4da2bdf77f5be7926c43dcbf2c765aa0cf30dcc150b32cfdc8c701b021ab"} Oct 30 13:32:52.628485 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:52.628432 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7bd69dd-m7lbc" podStartSLOduration=1.628417258 podStartE2EDuration="1.628417258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:52.625214127 +0000 UTC m=+2871.228712576" watchObservedRunningTime="2026-10-30 13:32:52.628417258 +0000 UTC m=+2871.231915704" Oct 30 13:32:53.591529 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:53.591497 4931 generic.go:356] "Generic (PLEG): container finished" podID="676089c8-e20a-4aa2-8220-be872d92bead" containerID="20d5cf0f619190ff7dc8ed463717b550ae60a828e9fbc10a37887a50704e27af" exitCode=0 Oct 30 13:32:53.591935 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:53.591542 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7bd69dd-m7lbc" event={"ID":"676089c8-e20a-4aa2-8220-be872d92bead","Type":"ContainerDied","Data":"20d5cf0f619190ff7dc8ed463717b550ae60a828e9fbc10a37887a50704e27af"} Oct 30 13:32:54.777239 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:54.777213 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7bd69dd-m7lbc" Oct 30 13:32:54.879532 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:54.879442 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/676089c8-e20a-4aa2-8220-be872d92bead-kube-api-access-zntq7\" (UniqueName: \"kubernetes.io/projected/676089c8-e20a-4aa2-8220-be872d92bead-kube-api-access-zntq7\") pod \"676089c8-e20a-4aa2-8220-be872d92bead\" (UID: \"676089c8-e20a-4aa2-8220-be872d92bead\") " Oct 30 13:32:54.879695 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:54.879532 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/676089c8-e20a-4aa2-8220-be872d92bead-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/676089c8-e20a-4aa2-8220-be872d92bead-maas-api-xks-ca\") pod \"676089c8-e20a-4aa2-8220-be872d92bead\" (UID: \"676089c8-e20a-4aa2-8220-be872d92bead\") " Oct 30 13:32:54.879747 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:54.879686 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/676089c8-e20a-4aa2-8220-be872d92bead-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/676089c8-e20a-4aa2-8220-be872d92bead-maas-api-openshift-ca\") pod \"676089c8-e20a-4aa2-8220-be872d92bead\" (UID: \"676089c8-e20a-4aa2-8220-be872d92bead\") " Oct 30 13:32:54.880247 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:54.880211 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/676089c8-e20a-4aa2-8220-be872d92bead-maas-api-openshift-ca" pod "676089c8-e20a-4aa2-8220-be872d92bead" (UID: "676089c8-e20a-4aa2-8220-be872d92bead"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:32:54.880650 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:54.880625 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/676089c8-e20a-4aa2-8220-be872d92bead-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:32:54.882231 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:54.882205 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/676089c8-e20a-4aa2-8220-be872d92bead-kube-api-access-zntq7" pod "676089c8-e20a-4aa2-8220-be872d92bead" (UID: "676089c8-e20a-4aa2-8220-be872d92bead"). InnerVolumeSpecName "kube-api-access-zntq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:32:54.882415 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:54.882202 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/676089c8-e20a-4aa2-8220-be872d92bead-maas-api-xks-ca" pod "676089c8-e20a-4aa2-8220-be872d92bead" (UID: "676089c8-e20a-4aa2-8220-be872d92bead"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:32:54.982512 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:54.982478 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zntq7\" (UniqueName: \"kubernetes.io/projected/676089c8-e20a-4aa2-8220-be872d92bead-kube-api-access-zntq7\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:32:54.982512 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:54.982508 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/676089c8-e20a-4aa2-8220-be872d92bead-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:32:55.615951 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:55.612327 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7bd69dd-m7lbc" event={"ID":"676089c8-e20a-4aa2-8220-be872d92bead","Type":"ContainerDied","Data":"361b4da2bdf77f5be7926c43dcbf2c765aa0cf30dcc150b32cfdc8c701b021ab"} Oct 30 13:32:55.615951 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:55.612450 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="361b4da2bdf77f5be7926c43dcbf2c765aa0cf30dcc150b32cfdc8c701b021ab" Oct 30 13:32:55.615951 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:55.612548 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7bd69dd-m7lbc" Oct 30 13:32:56.530021 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:56.529989 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq"] Oct 30 13:32:56.530676 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:56.530657 4931 cpu_manager.go:414] "containerMap: removing container" podUID="676089c8-e20a-4aa2-8220-be872d92bead" containerName="revoke-keys" Oct 30 13:32:56.530752 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:56.530678 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="676089c8-e20a-4aa2-8220-be872d92bead" containerName="revoke-keys" Oct 30 13:32:56.530838 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:56.530829 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="676089c8-e20a-4aa2-8220-be872d92bead" containerName="revoke-keys" Oct 30 13:32:56.534718 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:56.534695 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" Oct 30 13:32:56.573185 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:56.573156 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq"] Oct 30 13:32:56.607691 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:56.607651 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d987afd-6dc8-401b-a339-3909dff59665-istiod-ca-cert\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" Oct 30 13:32:56.607864 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:56.607704 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d987afd-6dc8-401b-a339-3909dff59665-istio-data\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" Oct 30 13:32:56.607864 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:56.607820 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d987afd-6dc8-401b-a339-3909dff59665-istio-podinfo\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" Oct 30 13:32:56.607960 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:56.607870 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d987afd-6dc8-401b-a339-3909dff59665-istio-token\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" Oct 30 13:32:56.607960 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:56.607894 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d987afd-6dc8-401b-a339-3909dff59665-istio-envoy\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" Oct 30 13:32:56.607960 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:56.607947 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ljbc\" (UniqueName: \"kubernetes.io/projected/6d987afd-6dc8-401b-a339-3909dff59665-kube-api-access-4ljbc\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" Oct 30 13:32:56.608079 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:56.608045 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d987afd-6dc8-401b-a339-3909dff59665-workload-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" Oct 30 13:32:56.608137 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:56.608115 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d987afd-6dc8-401b-a339-3909dff59665-workload-certs\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" Oct 30 13:32:56.608194 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:56.608177 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d987afd-6dc8-401b-a339-3909dff59665-credential-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" Oct 30 13:32:56.709708 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:56.709673 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d987afd-6dc8-401b-a339-3909dff59665-istio-podinfo\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" Oct 30 13:32:56.709928 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:56.709729 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d987afd-6dc8-401b-a339-3909dff59665-istio-token\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" Oct 30 13:32:56.709928 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:56.709757 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d987afd-6dc8-401b-a339-3909dff59665-istio-envoy\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" Oct 30 13:32:56.709928 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:56.709806 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ljbc\" (UniqueName: \"kubernetes.io/projected/6d987afd-6dc8-401b-a339-3909dff59665-kube-api-access-4ljbc\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" Oct 30 13:32:56.709928 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:56.709905 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d987afd-6dc8-401b-a339-3909dff59665-workload-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" Oct 30 13:32:56.710167 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:56.709996 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d987afd-6dc8-401b-a339-3909dff59665-workload-certs\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" Oct 30 13:32:56.710291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:56.710249 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d987afd-6dc8-401b-a339-3909dff59665-credential-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" Oct 30 13:32:56.710291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:56.710291 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d987afd-6dc8-401b-a339-3909dff59665-workload-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" Oct 30 13:32:56.710507 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:56.710454 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d987afd-6dc8-401b-a339-3909dff59665-istiod-ca-cert\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" Oct 30 13:32:56.710507 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:56.710459 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d987afd-6dc8-401b-a339-3909dff59665-workload-certs\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" Oct 30 13:32:56.710507 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:56.710487 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d987afd-6dc8-401b-a339-3909dff59665-istio-data\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" Oct 30 13:32:56.710880 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:56.710855 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d987afd-6dc8-401b-a339-3909dff59665-istio-data\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" Oct 30 13:32:56.710948 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:56.710881 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d987afd-6dc8-401b-a339-3909dff59665-credential-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" Oct 30 13:32:56.711075 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:56.711058 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d987afd-6dc8-401b-a339-3909dff59665-istiod-ca-cert\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" Oct 30 13:32:56.712277 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:56.712246 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d987afd-6dc8-401b-a339-3909dff59665-istio-envoy\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" Oct 30 13:32:56.712518 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:56.712498 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d987afd-6dc8-401b-a339-3909dff59665-istio-podinfo\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" Oct 30 13:32:56.734455 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:56.734433 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d987afd-6dc8-401b-a339-3909dff59665-istio-token\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" Oct 30 13:32:56.734645 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:56.734623 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ljbc\" (UniqueName: \"kubernetes.io/projected/6d987afd-6dc8-401b-a339-3909dff59665-kube-api-access-4ljbc\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" Oct 30 13:32:56.849444 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:56.849369 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" Oct 30 13:32:57.183320 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:32:57.183290 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d987afd_6dc8_401b_a339_3909dff59665.slice/crio-cbe618eff40eca857647eaad99b05b141a55c622118b490cc8c3446b095363bf WatchSource:0}: Error finding container cbe618eff40eca857647eaad99b05b141a55c622118b490cc8c3446b095363bf: Status 404 returned error can't find the container with id cbe618eff40eca857647eaad99b05b141a55c622118b490cc8c3446b095363bf Oct 30 13:32:57.183772 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:57.183749 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq"] Oct 30 13:32:57.186293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:57.186244 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:32:57.186379 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:57.186334 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:32:57.186379 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:57.186367 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:32:57.368193 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:57.368149 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1981ad18-r7phw"] Oct 30 13:32:57.373683 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:57.373658 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1981ad18-r7phw" Oct 30 13:32:57.398308 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:57.391613 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:32:57.398308 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:57.391841 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:32:57.398308 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:57.397583 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1981ad18-r7phw"] Oct 30 13:32:57.523685 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:57.523646 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ca9233-afe0-4d73-b7dc-f969b928fdcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1981ad18-r7phw\" (UID: \"83ca9233-afe0-4d73-b7dc-f969b928fdcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1981ad18-r7phw" Oct 30 13:32:57.523890 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:57.523853 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ca9233-afe0-4d73-b7dc-f969b928fdcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1981ad18-r7phw\" (UID: \"83ca9233-afe0-4d73-b7dc-f969b928fdcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1981ad18-r7phw" Oct 30 13:32:57.524053 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:57.524036 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gw7k\" (UniqueName: \"kubernetes.io/projected/83ca9233-afe0-4d73-b7dc-f969b928fdcb-kube-api-access-7gw7k\") pod \"maas-api-revoke-sub-1981ad18-r7phw\" (UID: \"83ca9233-afe0-4d73-b7dc-f969b928fdcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1981ad18-r7phw" Oct 30 13:32:57.625374 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:57.625336 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ca9233-afe0-4d73-b7dc-f969b928fdcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1981ad18-r7phw\" (UID: \"83ca9233-afe0-4d73-b7dc-f969b928fdcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1981ad18-r7phw" Oct 30 13:32:57.625891 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:57.625566 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gw7k\" (UniqueName: \"kubernetes.io/projected/83ca9233-afe0-4d73-b7dc-f969b928fdcb-kube-api-access-7gw7k\") pod \"maas-api-revoke-sub-1981ad18-r7phw\" (UID: \"83ca9233-afe0-4d73-b7dc-f969b928fdcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1981ad18-r7phw" Oct 30 13:32:57.625891 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:57.625699 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ca9233-afe0-4d73-b7dc-f969b928fdcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1981ad18-r7phw\" (UID: \"83ca9233-afe0-4d73-b7dc-f969b928fdcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1981ad18-r7phw" Oct 30 13:32:57.626229 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:57.626197 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ca9233-afe0-4d73-b7dc-f969b928fdcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1981ad18-r7phw\" (UID: \"83ca9233-afe0-4d73-b7dc-f969b928fdcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1981ad18-r7phw" Oct 30 13:32:57.628725 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:57.628701 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ca9233-afe0-4d73-b7dc-f969b928fdcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1981ad18-r7phw\" (UID: \"83ca9233-afe0-4d73-b7dc-f969b928fdcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1981ad18-r7phw" Oct 30 13:32:57.635096 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:57.635060 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" event={"ID":"6d987afd-6dc8-401b-a339-3909dff59665","Type":"ContainerStarted","Data":"4bdcfb97e3fe87e7eea3b0d9d44cea2ede26f7e1db3835cf001a958f272cf289"} Oct 30 13:32:57.635249 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:57.635130 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" event={"ID":"6d987afd-6dc8-401b-a339-3909dff59665","Type":"ContainerStarted","Data":"cbe618eff40eca857647eaad99b05b141a55c622118b490cc8c3446b095363bf"} Oct 30 13:32:57.650534 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:57.650513 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gw7k\" (UniqueName: \"kubernetes.io/projected/83ca9233-afe0-4d73-b7dc-f969b928fdcb-kube-api-access-7gw7k\") pod \"maas-api-revoke-sub-1981ad18-r7phw\" (UID: \"83ca9233-afe0-4d73-b7dc-f969b928fdcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1981ad18-r7phw" Oct 30 13:32:57.688855 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:57.688800 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" podStartSLOduration=1.6887865450000001 podStartE2EDuration="1.688786545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:57.686933629 +0000 UTC m=+2876.290432078" watchObservedRunningTime="2026-10-30 13:32:57.688786545 +0000 UTC m=+2876.292284990" Oct 30 13:32:57.691816 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:57.691787 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1981ad18-r7phw" Oct 30 13:32:57.849979 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:57.849897 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" Oct 30 13:32:58.006791 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:58.006758 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1981ad18-r7phw"] Oct 30 13:32:58.007228 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:32:58.007202 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ca9233_afe0_4d73_b7dc_f969b928fdcb.slice/crio-de70eca0a99066178fa03fdfe86887f949512bb32c4ca8a9b845d61bade7e785 WatchSource:0}: Error finding container de70eca0a99066178fa03fdfe86887f949512bb32c4ca8a9b845d61bade7e785: Status 404 returned error can't find the container with id de70eca0a99066178fa03fdfe86887f949512bb32c4ca8a9b845d61bade7e785 Oct 30 13:32:58.667105 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:58.667005 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1981ad18-r7phw" event={"ID":"83ca9233-afe0-4d73-b7dc-f969b928fdcb","Type":"ContainerStarted","Data":"14fa7bf0d4face1d856677ad9c6ac64211154f7bd1c11e56050677a2c3c873e6"} Oct 30 13:32:58.667105 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:58.667078 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1981ad18-r7phw" event={"ID":"83ca9233-afe0-4d73-b7dc-f969b928fdcb","Type":"ContainerStarted","Data":"de70eca0a99066178fa03fdfe86887f949512bb32c4ca8a9b845d61bade7e785"} Oct 30 13:32:58.712606 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:58.712532 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1981ad18-r7phw" podStartSLOduration=1.712510904 podStartE2EDuration="1.712510904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:58.709941269 +0000 UTC m=+2877.313439714" watchObservedRunningTime="2026-10-30 13:32:58.712510904 +0000 UTC m=+2877.316009354" Oct 30 13:32:58.874338 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:58.870401 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 13:32:58.874338 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:58.870472 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" podUID="6d987afd-6dc8-401b-a339-3909dff59665" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 13:32:59.682688 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:59.682653 4931 generic.go:356] "Generic (PLEG): container finished" podID="83ca9233-afe0-4d73-b7dc-f969b928fdcb" containerID="14fa7bf0d4face1d856677ad9c6ac64211154f7bd1c11e56050677a2c3c873e6" exitCode=0 Oct 30 13:32:59.683140 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:59.682741 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1981ad18-r7phw" event={"ID":"83ca9233-afe0-4d73-b7dc-f969b928fdcb","Type":"ContainerDied","Data":"14fa7bf0d4face1d856677ad9c6ac64211154f7bd1c11e56050677a2c3c873e6"} Oct 30 13:32:59.859486 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:59.859428 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 13:32:59.859745 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:32:59.859499 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" podUID="6d987afd-6dc8-401b-a339-3909dff59665" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 13:33:00.876861 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:00.876821 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 13:33:00.877390 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:00.876889 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" podUID="6d987afd-6dc8-401b-a339-3909dff59665" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 13:33:00.914953 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:00.914927 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1981ad18-r7phw" Oct 30 13:33:01.029016 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:01.028935 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83ca9233-afe0-4d73-b7dc-f969b928fdcb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ca9233-afe0-4d73-b7dc-f969b928fdcb-maas-api-openshift-ca\") pod \"83ca9233-afe0-4d73-b7dc-f969b928fdcb\" (UID: \"83ca9233-afe0-4d73-b7dc-f969b928fdcb\") " Oct 30 13:33:01.029177 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:01.029032 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83ca9233-afe0-4d73-b7dc-f969b928fdcb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ca9233-afe0-4d73-b7dc-f969b928fdcb-maas-api-xks-ca\") pod \"83ca9233-afe0-4d73-b7dc-f969b928fdcb\" (UID: \"83ca9233-afe0-4d73-b7dc-f969b928fdcb\") " Oct 30 13:33:01.029316 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:01.029178 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83ca9233-afe0-4d73-b7dc-f969b928fdcb-kube-api-access-7gw7k\" (UniqueName: \"kubernetes.io/projected/83ca9233-afe0-4d73-b7dc-f969b928fdcb-kube-api-access-7gw7k\") pod \"83ca9233-afe0-4d73-b7dc-f969b928fdcb\" (UID: \"83ca9233-afe0-4d73-b7dc-f969b928fdcb\") " Oct 30 13:33:01.029316 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:01.029275 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83ca9233-afe0-4d73-b7dc-f969b928fdcb-maas-api-openshift-ca" pod "83ca9233-afe0-4d73-b7dc-f969b928fdcb" (UID: "83ca9233-afe0-4d73-b7dc-f969b928fdcb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:33:01.029704 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:01.029686 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ca9233-afe0-4d73-b7dc-f969b928fdcb-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:33:01.031179 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:01.031159 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83ca9233-afe0-4d73-b7dc-f969b928fdcb-maas-api-xks-ca" pod "83ca9233-afe0-4d73-b7dc-f969b928fdcb" (UID: "83ca9233-afe0-4d73-b7dc-f969b928fdcb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:33:01.031506 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:01.031483 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83ca9233-afe0-4d73-b7dc-f969b928fdcb-kube-api-access-7gw7k" pod "83ca9233-afe0-4d73-b7dc-f969b928fdcb" (UID: "83ca9233-afe0-4d73-b7dc-f969b928fdcb"). InnerVolumeSpecName "kube-api-access-7gw7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:33:01.131471 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:01.131439 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gw7k\" (UniqueName: \"kubernetes.io/projected/83ca9233-afe0-4d73-b7dc-f969b928fdcb-kube-api-access-7gw7k\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:33:01.131471 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:01.131468 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ca9233-afe0-4d73-b7dc-f969b928fdcb-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:33:01.662530 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:01.662495 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fc55545-bwn9x"] Oct 30 13:33:01.663151 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:01.663135 4931 cpu_manager.go:414] "containerMap: removing container" podUID="83ca9233-afe0-4d73-b7dc-f969b928fdcb" containerName="revoke-keys" Oct 30 13:33:01.663199 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:01.663153 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83ca9233-afe0-4d73-b7dc-f969b928fdcb" containerName="revoke-keys" Oct 30 13:33:01.663294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:01.663277 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="83ca9233-afe0-4d73-b7dc-f969b928fdcb" containerName="revoke-keys" Oct 30 13:33:01.668423 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:01.668395 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc55545-bwn9x" Oct 30 13:33:01.685278 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:01.685238 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fc55545-bwn9x"] Oct 30 13:33:01.706629 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:01.706599 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1981ad18-r7phw" Oct 30 13:33:01.706791 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:01.706595 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1981ad18-r7phw" event={"ID":"83ca9233-afe0-4d73-b7dc-f969b928fdcb","Type":"ContainerDied","Data":"de70eca0a99066178fa03fdfe86887f949512bb32c4ca8a9b845d61bade7e785"} Oct 30 13:33:01.706791 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:01.706726 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de70eca0a99066178fa03fdfe86887f949512bb32c4ca8a9b845d61bade7e785" Oct 30 13:33:01.761239 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:01.761201 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/186079bb-e477-4fee-b293-54356a7e6310-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fc55545-bwn9x\" (UID: \"186079bb-e477-4fee-b293-54356a7e6310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc55545-bwn9x" Oct 30 13:33:01.761455 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:01.761314 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/186079bb-e477-4fee-b293-54356a7e6310-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fc55545-bwn9x\" (UID: \"186079bb-e477-4fee-b293-54356a7e6310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc55545-bwn9x" Oct 30 13:33:01.761543 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:01.761482 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5v4l\" (UniqueName: \"kubernetes.io/projected/186079bb-e477-4fee-b293-54356a7e6310-kube-api-access-w5v4l\") pod \"maas-api-revoke-sub-3fc55545-bwn9x\" (UID: \"186079bb-e477-4fee-b293-54356a7e6310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc55545-bwn9x" Oct 30 13:33:01.857343 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:01.856690 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 13:33:01.857343 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:01.856769 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" podUID="6d987afd-6dc8-401b-a339-3909dff59665" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 13:33:01.863290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:01.863086 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/186079bb-e477-4fee-b293-54356a7e6310-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fc55545-bwn9x\" (UID: \"186079bb-e477-4fee-b293-54356a7e6310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc55545-bwn9x" Oct 30 13:33:01.863290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:01.863156 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/186079bb-e477-4fee-b293-54356a7e6310-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fc55545-bwn9x\" (UID: \"186079bb-e477-4fee-b293-54356a7e6310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc55545-bwn9x" Oct 30 13:33:01.863290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:01.863245 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5v4l\" (UniqueName: \"kubernetes.io/projected/186079bb-e477-4fee-b293-54356a7e6310-kube-api-access-w5v4l\") pod \"maas-api-revoke-sub-3fc55545-bwn9x\" (UID: \"186079bb-e477-4fee-b293-54356a7e6310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc55545-bwn9x" Oct 30 13:33:01.906289 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:01.885961 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/186079bb-e477-4fee-b293-54356a7e6310-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fc55545-bwn9x\" (UID: \"186079bb-e477-4fee-b293-54356a7e6310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc55545-bwn9x" Oct 30 13:33:01.906964 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:01.892106 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/186079bb-e477-4fee-b293-54356a7e6310-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fc55545-bwn9x\" (UID: \"186079bb-e477-4fee-b293-54356a7e6310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc55545-bwn9x" Oct 30 13:33:01.927112 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:01.927073 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5v4l\" (UniqueName: \"kubernetes.io/projected/186079bb-e477-4fee-b293-54356a7e6310-kube-api-access-w5v4l\") pod \"maas-api-revoke-sub-3fc55545-bwn9x\" (UID: \"186079bb-e477-4fee-b293-54356a7e6310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc55545-bwn9x" Oct 30 13:33:01.988008 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:01.987978 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc55545-bwn9x" Oct 30 13:33:02.327654 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:33:02.327615 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod186079bb_e477_4fee_b293_54356a7e6310.slice/crio-8c02ef2e0954eb7cc8cc64c516d67e0e3eeb063e2909226a0e86fabc211b5372 WatchSource:0}: Error finding container 8c02ef2e0954eb7cc8cc64c516d67e0e3eeb063e2909226a0e86fabc211b5372: Status 404 returned error can't find the container with id 8c02ef2e0954eb7cc8cc64c516d67e0e3eeb063e2909226a0e86fabc211b5372 Oct 30 13:33:02.328299 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:02.328164 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fc55545-bwn9x"] Oct 30 13:33:02.717216 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:02.717183 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc55545-bwn9x" event={"ID":"186079bb-e477-4fee-b293-54356a7e6310","Type":"ContainerStarted","Data":"8c02ef2e0954eb7cc8cc64c516d67e0e3eeb063e2909226a0e86fabc211b5372"} Oct 30 13:33:02.852292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:02.850172 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 13:33:02.852292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:02.850246 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" podUID="6d987afd-6dc8-401b-a339-3909dff59665" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 13:33:03.477936 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:03.477906 4931 scope.go:122] "RemoveContainer" containerID="3c020b3dbc9c2178cf0808d4f2a6fdfbcc9608b78ab8fc1de3ed88b5ffb2f42b" Oct 30 13:33:03.564972 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:03.564947 4931 scope.go:122] "RemoveContainer" containerID="c135dd7d09767db66e07ab2581bb159c0905d432a0a7f81f4a3b6c3a40d3107d" Oct 30 13:33:03.728349 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:03.728215 4931 generic.go:356] "Generic (PLEG): container finished" podID="186079bb-e477-4fee-b293-54356a7e6310" containerID="bff679f4cf90770700d94def65e70afa21b10a539c50e38078591749a8784758" exitCode=0 Oct 30 13:33:03.728349 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:03.728298 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc55545-bwn9x" event={"ID":"186079bb-e477-4fee-b293-54356a7e6310","Type":"ContainerDied","Data":"bff679f4cf90770700d94def65e70afa21b10a539c50e38078591749a8784758"} Oct 30 13:33:03.857295 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:03.850958 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 13:33:03.857295 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:03.851022 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" podUID="6d987afd-6dc8-401b-a339-3909dff59665" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 13:33:04.855656 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:04.855623 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" Oct 30 13:33:04.856348 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:04.856321 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" Oct 30 13:33:04.857353 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:04.857330 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" Oct 30 13:33:04.911172 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:04.911145 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc55545-bwn9x" Oct 30 13:33:05.031643 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:05.031540 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/186079bb-e477-4fee-b293-54356a7e6310-kube-api-access-w5v4l\" (UniqueName: \"kubernetes.io/projected/186079bb-e477-4fee-b293-54356a7e6310-kube-api-access-w5v4l\") pod \"186079bb-e477-4fee-b293-54356a7e6310\" (UID: \"186079bb-e477-4fee-b293-54356a7e6310\") " Oct 30 13:33:05.031643 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:05.031611 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/186079bb-e477-4fee-b293-54356a7e6310-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/186079bb-e477-4fee-b293-54356a7e6310-maas-api-xks-ca\") pod \"186079bb-e477-4fee-b293-54356a7e6310\" (UID: \"186079bb-e477-4fee-b293-54356a7e6310\") " Oct 30 13:33:05.031897 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:05.031826 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/186079bb-e477-4fee-b293-54356a7e6310-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/186079bb-e477-4fee-b293-54356a7e6310-maas-api-openshift-ca\") pod \"186079bb-e477-4fee-b293-54356a7e6310\" (UID: \"186079bb-e477-4fee-b293-54356a7e6310\") " Oct 30 13:33:05.032318 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:05.032242 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/186079bb-e477-4fee-b293-54356a7e6310-maas-api-openshift-ca" pod "186079bb-e477-4fee-b293-54356a7e6310" (UID: "186079bb-e477-4fee-b293-54356a7e6310"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:33:05.032995 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:05.032972 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/186079bb-e477-4fee-b293-54356a7e6310-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:33:05.034655 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:05.034619 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/186079bb-e477-4fee-b293-54356a7e6310-maas-api-xks-ca" pod "186079bb-e477-4fee-b293-54356a7e6310" (UID: "186079bb-e477-4fee-b293-54356a7e6310"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:33:05.034765 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:05.034672 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/186079bb-e477-4fee-b293-54356a7e6310-kube-api-access-w5v4l" pod "186079bb-e477-4fee-b293-54356a7e6310" (UID: "186079bb-e477-4fee-b293-54356a7e6310"). InnerVolumeSpecName "kube-api-access-w5v4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:33:05.134994 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:05.134956 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5v4l\" (UniqueName: \"kubernetes.io/projected/186079bb-e477-4fee-b293-54356a7e6310-kube-api-access-w5v4l\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:33:05.134994 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:05.134992 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/186079bb-e477-4fee-b293-54356a7e6310-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:33:05.746281 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:05.746233 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc55545-bwn9x" Oct 30 13:33:05.746465 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:05.746254 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc55545-bwn9x" event={"ID":"186079bb-e477-4fee-b293-54356a7e6310","Type":"ContainerDied","Data":"8c02ef2e0954eb7cc8cc64c516d67e0e3eeb063e2909226a0e86fabc211b5372"} Oct 30 13:33:05.746465 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:05.746339 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c02ef2e0954eb7cc8cc64c516d67e0e3eeb063e2909226a0e86fabc211b5372" Oct 30 13:33:06.746423 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.746383 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4"] Oct 30 13:33:06.747848 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.747299 4931 cpu_manager.go:414] "containerMap: removing container" podUID="186079bb-e477-4fee-b293-54356a7e6310" containerName="revoke-keys" Oct 30 13:33:06.747848 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.747327 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="186079bb-e477-4fee-b293-54356a7e6310" containerName="revoke-keys" Oct 30 13:33:06.747848 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.747573 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="186079bb-e477-4fee-b293-54356a7e6310" containerName="revoke-keys" Oct 30 13:33:06.752501 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.752476 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" Oct 30 13:33:06.768659 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.768621 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv"] Oct 30 13:33:06.773810 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.773777 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" Oct 30 13:33:06.783135 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.783107 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4"] Oct 30 13:33:06.807858 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.807826 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv"] Oct 30 13:33:06.863530 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.863490 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a33526d-ab5b-4e65-8111-b5a398514d40-istiod-ca-cert\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" Oct 30 13:33:06.863742 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.863580 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a33526d-ab5b-4e65-8111-b5a398514d40-istio-podinfo\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" Oct 30 13:33:06.863891 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.863743 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a33526d-ab5b-4e65-8111-b5a398514d40-istio-data\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" Oct 30 13:33:06.863891 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.863788 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2776ed01-5f18-493e-bf9b-7fbb328e01a2-credential-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" Oct 30 13:33:06.864010 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.863892 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxfmv\" (UniqueName: \"kubernetes.io/projected/2776ed01-5f18-493e-bf9b-7fbb328e01a2-kube-api-access-cxfmv\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" Oct 30 13:33:06.864010 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.863961 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2776ed01-5f18-493e-bf9b-7fbb328e01a2-istio-podinfo\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" Oct 30 13:33:06.864124 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.864028 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a33526d-ab5b-4e65-8111-b5a398514d40-credential-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" Oct 30 13:33:06.864124 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.864046 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a33526d-ab5b-4e65-8111-b5a398514d40-istio-token\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" Oct 30 13:33:06.864124 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.864066 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2776ed01-5f18-493e-bf9b-7fbb328e01a2-istio-envoy\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" Oct 30 13:33:06.864337 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.864189 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2776ed01-5f18-493e-bf9b-7fbb328e01a2-istiod-ca-cert\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" Oct 30 13:33:06.864337 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.864211 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a33526d-ab5b-4e65-8111-b5a398514d40-workload-certs\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" Oct 30 13:33:06.864337 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.864317 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a33526d-ab5b-4e65-8111-b5a398514d40-workload-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" Oct 30 13:33:06.864538 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.864345 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2776ed01-5f18-493e-bf9b-7fbb328e01a2-workload-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" Oct 30 13:33:06.864538 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.864363 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a33526d-ab5b-4e65-8111-b5a398514d40-istio-envoy\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" Oct 30 13:33:06.864538 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.864381 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2776ed01-5f18-493e-bf9b-7fbb328e01a2-workload-certs\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" Oct 30 13:33:06.864538 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.864410 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdk77\" (UniqueName: \"kubernetes.io/projected/7a33526d-ab5b-4e65-8111-b5a398514d40-kube-api-access-fdk77\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" Oct 30 13:33:06.864538 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.864430 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2776ed01-5f18-493e-bf9b-7fbb328e01a2-istio-data\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" Oct 30 13:33:06.864538 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.864464 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2776ed01-5f18-493e-bf9b-7fbb328e01a2-istio-token\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" Oct 30 13:33:06.965548 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.965508 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxfmv\" (UniqueName: \"kubernetes.io/projected/2776ed01-5f18-493e-bf9b-7fbb328e01a2-kube-api-access-cxfmv\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" Oct 30 13:33:06.965548 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.965553 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2776ed01-5f18-493e-bf9b-7fbb328e01a2-istio-podinfo\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" Oct 30 13:33:06.965849 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.965600 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a33526d-ab5b-4e65-8111-b5a398514d40-credential-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" Oct 30 13:33:06.965849 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.965617 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a33526d-ab5b-4e65-8111-b5a398514d40-istio-token\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" Oct 30 13:33:06.965849 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.965636 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2776ed01-5f18-493e-bf9b-7fbb328e01a2-istio-envoy\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" Oct 30 13:33:06.965849 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.965694 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2776ed01-5f18-493e-bf9b-7fbb328e01a2-istiod-ca-cert\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" Oct 30 13:33:06.965849 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.965720 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a33526d-ab5b-4e65-8111-b5a398514d40-workload-certs\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" Oct 30 13:33:06.965849 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.965747 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a33526d-ab5b-4e65-8111-b5a398514d40-workload-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" Oct 30 13:33:06.965849 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.965779 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2776ed01-5f18-493e-bf9b-7fbb328e01a2-workload-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" Oct 30 13:33:06.965849 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.965798 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a33526d-ab5b-4e65-8111-b5a398514d40-istio-envoy\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" Oct 30 13:33:06.965849 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.965823 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2776ed01-5f18-493e-bf9b-7fbb328e01a2-workload-certs\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" Oct 30 13:33:06.966366 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.965866 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdk77\" (UniqueName: \"kubernetes.io/projected/7a33526d-ab5b-4e65-8111-b5a398514d40-kube-api-access-fdk77\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" Oct 30 13:33:06.966366 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.965897 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2776ed01-5f18-493e-bf9b-7fbb328e01a2-istio-data\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" Oct 30 13:33:06.966366 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.965948 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2776ed01-5f18-493e-bf9b-7fbb328e01a2-istio-token\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" Oct 30 13:33:06.966366 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.966030 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a33526d-ab5b-4e65-8111-b5a398514d40-istiod-ca-cert\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" Oct 30 13:33:06.966366 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.966099 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a33526d-ab5b-4e65-8111-b5a398514d40-istio-podinfo\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" Oct 30 13:33:06.966366 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.966182 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a33526d-ab5b-4e65-8111-b5a398514d40-istio-data\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" Oct 30 13:33:06.966366 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.966215 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2776ed01-5f18-493e-bf9b-7fbb328e01a2-credential-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" Oct 30 13:33:06.966366 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.966239 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2776ed01-5f18-493e-bf9b-7fbb328e01a2-workload-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" Oct 30 13:33:06.966366 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.966338 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2776ed01-5f18-493e-bf9b-7fbb328e01a2-workload-certs\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" Oct 30 13:33:06.966366 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.966359 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a33526d-ab5b-4e65-8111-b5a398514d40-credential-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" Oct 30 13:33:06.966938 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.966606 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a33526d-ab5b-4e65-8111-b5a398514d40-workload-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" Oct 30 13:33:06.966938 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.966603 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2776ed01-5f18-493e-bf9b-7fbb328e01a2-istiod-ca-cert\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" Oct 30 13:33:06.966938 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.966606 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2776ed01-5f18-493e-bf9b-7fbb328e01a2-istio-data\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" Oct 30 13:33:06.966938 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.966852 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a33526d-ab5b-4e65-8111-b5a398514d40-workload-certs\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" Oct 30 13:33:06.966938 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.966881 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2776ed01-5f18-493e-bf9b-7fbb328e01a2-credential-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" Oct 30 13:33:06.967221 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.966996 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a33526d-ab5b-4e65-8111-b5a398514d40-istiod-ca-cert\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" Oct 30 13:33:06.967221 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.967056 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a33526d-ab5b-4e65-8111-b5a398514d40-istio-data\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" Oct 30 13:33:06.968451 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.968427 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2776ed01-5f18-493e-bf9b-7fbb328e01a2-istio-envoy\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" Oct 30 13:33:06.968804 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.968777 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2776ed01-5f18-493e-bf9b-7fbb328e01a2-istio-podinfo\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" Oct 30 13:33:06.968946 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.968869 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a33526d-ab5b-4e65-8111-b5a398514d40-istio-podinfo\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" Oct 30 13:33:06.969015 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.968963 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a33526d-ab5b-4e65-8111-b5a398514d40-istio-envoy\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" Oct 30 13:33:06.992907 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.992883 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2776ed01-5f18-493e-bf9b-7fbb328e01a2-istio-token\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" Oct 30 13:33:06.993518 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.993500 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a33526d-ab5b-4e65-8111-b5a398514d40-istio-token\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" Oct 30 13:33:06.998202 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.998144 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdk77\" (UniqueName: \"kubernetes.io/projected/7a33526d-ab5b-4e65-8111-b5a398514d40-kube-api-access-fdk77\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" Oct 30 13:33:06.998552 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:06.998531 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxfmv\" (UniqueName: \"kubernetes.io/projected/2776ed01-5f18-493e-bf9b-7fbb328e01a2-kube-api-access-cxfmv\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" Oct 30 13:33:07.074523 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:07.074486 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" Oct 30 13:33:07.109431 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:07.109364 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" Oct 30 13:33:07.355159 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:33:07.355076 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:33:07.409580 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:33:07.409551 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2776ed01_5f18_493e_bf9b_7fbb328e01a2.slice/crio-502deb6ac88a3570a10e2f405450bbf3c366901d8495e90e8c7945094faf30cb WatchSource:0}: Error finding container 502deb6ac88a3570a10e2f405450bbf3c366901d8495e90e8c7945094faf30cb: Status 404 returned error can't find the container with id 502deb6ac88a3570a10e2f405450bbf3c366901d8495e90e8c7945094faf30cb Oct 30 13:33:07.410788 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:07.410759 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4"] Oct 30 13:33:07.412562 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:07.412529 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:33:07.412634 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:07.412594 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:33:07.412634 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:07.412625 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:33:07.455387 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:33:07.455353 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a33526d_ab5b_4e65_8111_b5a398514d40.slice/crio-1c32cf5cc2ce7d2a2b7917713fa9dd3e387feb1ce3edf3f31fc6afb659a458c8 WatchSource:0}: Error finding container 1c32cf5cc2ce7d2a2b7917713fa9dd3e387feb1ce3edf3f31fc6afb659a458c8: Status 404 returned error can't find the container with id 1c32cf5cc2ce7d2a2b7917713fa9dd3e387feb1ce3edf3f31fc6afb659a458c8 Oct 30 13:33:07.457142 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:07.457116 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv"] Oct 30 13:33:07.460923 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:07.460867 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:33:07.461059 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:07.461036 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:33:07.461153 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:07.461131 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:33:07.768205 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:07.768167 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" event={"ID":"2776ed01-5f18-493e-bf9b-7fbb328e01a2","Type":"ContainerStarted","Data":"abe9d4324eceb24becf64e846066d1bd10981698ac0970ab00f5089dd831a6e5"} Oct 30 13:33:07.768807 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:07.768225 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" event={"ID":"2776ed01-5f18-493e-bf9b-7fbb328e01a2","Type":"ContainerStarted","Data":"502deb6ac88a3570a10e2f405450bbf3c366901d8495e90e8c7945094faf30cb"} Oct 30 13:33:07.770342 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:07.770303 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" event={"ID":"7a33526d-ab5b-4e65-8111-b5a398514d40","Type":"ContainerStarted","Data":"523ec54bf6530048c5b57739eb0d3f8a12bc5af74b132f6122f9796a282b195e"} Oct 30 13:33:07.770506 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:07.770372 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" event={"ID":"7a33526d-ab5b-4e65-8111-b5a398514d40","Type":"ContainerStarted","Data":"1c32cf5cc2ce7d2a2b7917713fa9dd3e387feb1ce3edf3f31fc6afb659a458c8"} Oct 30 13:33:07.817466 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:07.817407 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" podStartSLOduration=1.817392532 podStartE2EDuration="1.817392532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:07.815937375 +0000 UTC m=+2886.419435822" watchObservedRunningTime="2026-10-30 13:33:07.817392532 +0000 UTC m=+2886.420890977" Oct 30 13:33:07.866700 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:07.866641 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" podStartSLOduration=1.8666265709999998 podStartE2EDuration="1.866626571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:07.865082731 +0000 UTC m=+2886.468581213" watchObservedRunningTime="2026-10-30 13:33:07.866626571 +0000 UTC m=+2886.470125017" Oct 30 13:33:08.075820 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:08.075682 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" Oct 30 13:33:08.082316 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:08.081936 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 13:33:08.082316 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:08.081998 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" podUID="2776ed01-5f18-493e-bf9b-7fbb328e01a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 13:33:08.110096 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:08.110068 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" Oct 30 13:33:08.111782 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:08.111753 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 13:33:08.111911 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:08.111803 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" podUID="7a33526d-ab5b-4e65-8111-b5a398514d40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 13:33:09.076291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:09.075718 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 13:33:09.076291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:09.075787 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" podUID="2776ed01-5f18-493e-bf9b-7fbb328e01a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 13:33:09.110742 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:09.110700 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 13:33:09.110941 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:09.110746 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" podUID="7a33526d-ab5b-4e65-8111-b5a398514d40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 13:33:10.086321 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:10.083335 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 13:33:10.086321 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:10.083419 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" podUID="2776ed01-5f18-493e-bf9b-7fbb328e01a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 13:33:10.118382 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:10.116888 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 13:33:10.118382 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:10.116961 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" podUID="7a33526d-ab5b-4e65-8111-b5a398514d40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 13:33:11.076113 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:11.076066 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 13:33:11.076383 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:11.076131 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" podUID="2776ed01-5f18-493e-bf9b-7fbb328e01a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 13:33:11.110459 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:11.110424 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 13:33:11.110858 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:11.110488 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" podUID="7a33526d-ab5b-4e65-8111-b5a398514d40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 13:33:11.609293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:11.608284 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht"] Oct 30 13:33:12.128641 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.127471 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 13:33:12.128641 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.127542 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" podUID="7a33526d-ab5b-4e65-8111-b5a398514d40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 13:33:12.128641 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.128113 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 13:33:12.128641 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.128161 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" podUID="2776ed01-5f18-493e-bf9b-7fbb328e01a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 13:33:12.157357 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.142929 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" Oct 30 13:33:12.196406 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.173417 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht"] Oct 30 13:33:12.225865 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.225818 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-istiod-ca-cert\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" Oct 30 13:33:12.226183 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.226157 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-istio-envoy\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" Oct 30 13:33:12.226353 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.226336 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-istio-podinfo\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" Oct 30 13:33:12.226566 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.226545 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-istio-token\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" Oct 30 13:33:12.226736 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.226718 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-istio-data\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" Oct 30 13:33:12.226878 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.226860 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-workload-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" Oct 30 13:33:12.227057 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.227037 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-credential-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" Oct 30 13:33:12.227184 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.227171 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f65vq\" (UniqueName: \"kubernetes.io/projected/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-kube-api-access-f65vq\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" Oct 30 13:33:12.227314 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.227303 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-workload-certs\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" Oct 30 13:33:12.328424 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.328388 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-istio-envoy\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" Oct 30 13:33:12.328637 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.328443 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-istio-podinfo\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" Oct 30 13:33:12.328713 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.328637 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-istio-token\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" Oct 30 13:33:12.328713 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.328678 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-istio-data\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" Oct 30 13:33:12.328828 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.328709 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-workload-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" Oct 30 13:33:12.328828 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.328806 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-credential-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" Oct 30 13:33:12.328942 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.328850 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f65vq\" (UniqueName: \"kubernetes.io/projected/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-kube-api-access-f65vq\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" Oct 30 13:33:12.328942 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.328919 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-workload-certs\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" Oct 30 13:33:12.329053 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.329006 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-istiod-ca-cert\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" Oct 30 13:33:12.329237 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.329208 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-workload-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" Oct 30 13:33:12.329237 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.329226 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-istio-data\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" Oct 30 13:33:12.329439 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.329411 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-workload-certs\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" Oct 30 13:33:12.329541 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.329521 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-credential-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" Oct 30 13:33:12.329820 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.329797 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-istiod-ca-cert\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" Oct 30 13:33:12.366291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.331397 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-istio-envoy\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" Oct 30 13:33:12.366291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.331678 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-istio-podinfo\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" Oct 30 13:33:12.377566 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.377531 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f65vq\" (UniqueName: \"kubernetes.io/projected/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-kube-api-access-f65vq\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" Oct 30 13:33:12.385391 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.379615 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-istio-token\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" Oct 30 13:33:12.490077 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.489839 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" Oct 30 13:33:12.888505 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.888479 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht"] Oct 30 13:33:12.889157 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:33:12.889126 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf83a441_04d6_46bc_a81d_1c47f8a68e8e.slice/crio-aaaff346151a2fea891db93a452c42bcd2145f8825c893ab615a729f93ebbd5e WatchSource:0}: Error finding container aaaff346151a2fea891db93a452c42bcd2145f8825c893ab615a729f93ebbd5e: Status 404 returned error can't find the container with id aaaff346151a2fea891db93a452c42bcd2145f8825c893ab615a729f93ebbd5e Oct 30 13:33:12.893392 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.893358 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:33:12.893469 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.893430 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:33:12.893512 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:12.893463 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:33:13.083815 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:13.083777 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 13:33:13.084100 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:13.084070 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" podUID="2776ed01-5f18-493e-bf9b-7fbb328e01a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 13:33:13.110693 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:13.110656 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 13:33:13.111455 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:13.111417 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" podUID="7a33526d-ab5b-4e65-8111-b5a398514d40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 13:33:13.325610 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:13.324927 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" event={"ID":"cf83a441-04d6-46bc-a81d-1c47f8a68e8e","Type":"ContainerStarted","Data":"830343a1f2bbfbd0ee3a784909a82cacc2fd9005ffa15706b14d001645fb5816"} Oct 30 13:33:13.325610 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:13.324997 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" event={"ID":"cf83a441-04d6-46bc-a81d-1c47f8a68e8e","Type":"ContainerStarted","Data":"aaaff346151a2fea891db93a452c42bcd2145f8825c893ab615a729f93ebbd5e"} Oct 30 13:33:13.492424 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:13.492378 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" Oct 30 13:33:14.086668 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:14.086598 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" podStartSLOduration=3.086578173 podStartE2EDuration="3.086578173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:13.39487382 +0000 UTC m=+2891.998372268" watchObservedRunningTime="2026-10-30 13:33:14.086578173 +0000 UTC m=+2892.690076620" Oct 30 13:33:14.098291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:14.090535 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a132210-28hbg"] Oct 30 13:33:14.098291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:14.090851 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 13:33:14.098291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:14.090896 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" podUID="2776ed01-5f18-493e-bf9b-7fbb328e01a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 13:33:14.113424 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:14.113386 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 13:33:14.113692 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:14.113663 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" podUID="7a33526d-ab5b-4e65-8111-b5a398514d40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 13:33:14.128710 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:14.128677 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a132210-28hbg"] Oct 30 13:33:14.502290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:14.501471 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 30 13:33:14.502290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:14.501536 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" podUID="cf83a441-04d6-46bc-a81d-1c47f8a68e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 30 13:33:15.103578 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:15.103543 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d67fc6-tq2z2"] Oct 30 13:33:15.112161 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:15.112123 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d67fc6-tq2z2"] Oct 30 13:33:15.188669 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:15.186985 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" Oct 30 13:33:15.188669 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:15.188346 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" Oct 30 13:33:15.189259 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:15.189236 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" Oct 30 13:33:15.268293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:15.267557 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" Oct 30 13:33:15.269289 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:15.268928 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" Oct 30 13:33:15.273914 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:15.273891 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" Oct 30 13:33:15.491940 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:15.491483 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 30 13:33:15.491940 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:15.491545 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" podUID="cf83a441-04d6-46bc-a81d-1c47f8a68e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 30 13:33:15.527175 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:15.527136 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a612b14-5b32-4fc1-a656-2e2fab10530a" path="/var/lib/kubelet/pods/7a612b14-5b32-4fc1-a656-2e2fab10530a/volumes" Oct 30 13:33:15.527976 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:15.527952 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d62dd022-0695-4a89-8f8c-04d00f5dffed" path="/var/lib/kubelet/pods/d62dd022-0695-4a89-8f8c-04d00f5dffed/volumes" Oct 30 13:33:16.490722 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:16.490687 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 30 13:33:16.490914 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:16.490738 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" podUID="cf83a441-04d6-46bc-a81d-1c47f8a68e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 30 13:33:17.491207 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:17.491172 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 30 13:33:17.491638 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:17.491224 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" podUID="cf83a441-04d6-46bc-a81d-1c47f8a68e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 30 13:33:18.491519 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:18.491474 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 30 13:33:18.491922 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:18.491535 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" podUID="cf83a441-04d6-46bc-a81d-1c47f8a68e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 30 13:33:19.491035 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:19.490990 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 30 13:33:19.491245 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:19.491063 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" podUID="cf83a441-04d6-46bc-a81d-1c47f8a68e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 30 13:33:20.495660 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:20.495623 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" Oct 30 13:33:20.496182 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:20.496161 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" Oct 30 13:33:20.497376 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:20.497347 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" Oct 30 13:33:21.854323 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:21.843455 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79"] Oct 30 13:33:22.061774 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:22.061296 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" Oct 30 13:33:22.108457 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:22.108363 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79"] Oct 30 13:33:22.292924 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:22.292884 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-workload-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" Oct 30 13:33:22.293113 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:22.292937 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-istio-token\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" Oct 30 13:33:22.293113 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:22.292964 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ksjz\" (UniqueName: \"kubernetes.io/projected/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-kube-api-access-6ksjz\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" Oct 30 13:33:22.293210 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:22.293135 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-istio-podinfo\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" Oct 30 13:33:22.293275 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:22.293243 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-credential-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" Oct 30 13:33:22.293410 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:22.293390 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-istiod-ca-cert\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" Oct 30 13:33:22.293465 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:22.293423 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-workload-certs\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" Oct 30 13:33:22.293569 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:22.293555 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-istio-data\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" Oct 30 13:33:22.293615 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:22.293597 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-istio-envoy\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" Oct 30 13:33:22.395601 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:22.395507 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-istio-data\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" Oct 30 13:33:22.395601 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:22.395569 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-istio-envoy\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" Oct 30 13:33:22.395861 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:22.395640 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-workload-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" Oct 30 13:33:22.395861 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:22.395672 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-istio-token\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" Oct 30 13:33:22.395861 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:22.395693 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ksjz\" (UniqueName: \"kubernetes.io/projected/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-kube-api-access-6ksjz\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" Oct 30 13:33:22.395861 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:22.395740 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-istio-podinfo\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" Oct 30 13:33:22.395861 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:22.395806 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-credential-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" Oct 30 13:33:22.396151 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:22.395914 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-istiod-ca-cert\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" Oct 30 13:33:22.396151 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:22.395950 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-workload-certs\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" Oct 30 13:33:22.396151 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:22.395954 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-istio-data\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" Oct 30 13:33:22.396151 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:22.396072 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-workload-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" Oct 30 13:33:22.396395 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:22.396160 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-credential-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" Oct 30 13:33:22.396395 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:22.396256 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-workload-certs\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" Oct 30 13:33:22.396613 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:22.396591 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-istiod-ca-cert\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" Oct 30 13:33:22.398011 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:22.397987 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-istio-envoy\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" Oct 30 13:33:22.398420 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:22.398401 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-istio-podinfo\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" Oct 30 13:33:22.420607 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:22.420576 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ksjz\" (UniqueName: \"kubernetes.io/projected/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-kube-api-access-6ksjz\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" Oct 30 13:33:22.420745 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:22.420652 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-istio-token\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" Oct 30 13:33:22.446148 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:22.446114 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" Oct 30 13:33:22.778504 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:33:22.778469 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bc1a121_7dc1_4ccd_a395_68929a6f6ad1.slice/crio-82d5b8e5f9169727f72a1964ca2e3e6cc086612d66405896a902e37de6beb5bd WatchSource:0}: Error finding container 82d5b8e5f9169727f72a1964ca2e3e6cc086612d66405896a902e37de6beb5bd: Status 404 returned error can't find the container with id 82d5b8e5f9169727f72a1964ca2e3e6cc086612d66405896a902e37de6beb5bd Oct 30 13:33:22.779661 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:22.779631 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79"] Oct 30 13:33:22.782653 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:22.782605 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:33:22.782770 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:22.782702 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:33:22.782770 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:22.782745 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:33:23.684111 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:23.684071 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" event={"ID":"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1","Type":"ContainerStarted","Data":"233b07ef31af5a34f76e1a82d0b290c0ff3ca24a2a1093c0b05c79e175c5c1bd"} Oct 30 13:33:23.684680 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:23.684146 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" event={"ID":"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1","Type":"ContainerStarted","Data":"82d5b8e5f9169727f72a1964ca2e3e6cc086612d66405896a902e37de6beb5bd"} Oct 30 13:33:23.748033 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:23.747983 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" podStartSLOduration=2.747970369 podStartE2EDuration="2.747970369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:23.74494762 +0000 UTC m=+2902.348446066" watchObservedRunningTime="2026-10-30 13:33:23.747970369 +0000 UTC m=+2902.351468869" Oct 30 13:33:24.450292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:24.449966 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" Oct 30 13:33:24.460294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:24.451694 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 13:33:24.460294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:24.451765 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" podUID="8bc1a121-7dc1-4ccd-a395-68929a6f6ad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 13:33:25.110573 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:25.110535 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbbe51fe-9b99z"] Oct 30 13:33:25.131784 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:25.130494 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbbe51fe-9b99z"] Oct 30 13:33:25.458381 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:25.449985 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 13:33:25.458381 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:25.450044 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" podUID="8bc1a121-7dc1-4ccd-a395-68929a6f6ad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 13:33:25.515504 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:25.515469 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87e0e82a-63e8-4aca-bfb5-80ac6a235c5d" path="/var/lib/kubelet/pods/87e0e82a-63e8-4aca-bfb5-80ac6a235c5d/volumes" Oct 30 13:33:26.053289 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:26.050929 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1155b409-rwgjs"] Oct 30 13:33:26.063280 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:26.063230 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1155b409-rwgjs"] Oct 30 13:33:26.449650 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:26.449613 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 13:33:26.450258 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:26.450221 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" podUID="8bc1a121-7dc1-4ccd-a395-68929a6f6ad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 13:33:27.448291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:27.447670 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 13:33:27.448291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:27.447738 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" podUID="8bc1a121-7dc1-4ccd-a395-68929a6f6ad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 13:33:27.515537 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:27.515503 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20d5c662-db5a-49ac-acb7-4f8c094297fd" path="/var/lib/kubelet/pods/20d5c662-db5a-49ac-acb7-4f8c094297fd/volumes" Oct 30 13:33:28.457629 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:28.457578 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 13:33:28.457960 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:28.457916 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" podUID="8bc1a121-7dc1-4ccd-a395-68929a6f6ad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 13:33:29.452293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:29.449697 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 13:33:29.452293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:29.449777 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" podUID="8bc1a121-7dc1-4ccd-a395-68929a6f6ad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 13:33:30.452278 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:30.452242 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" Oct 30 13:33:30.452675 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:30.452644 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" Oct 30 13:33:30.453366 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:30.453343 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" Oct 30 13:33:32.098440 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:32.098403 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r"] Oct 30 13:33:37.051773 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:37.051734 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86bc23cf-ct5rg"] Oct 30 13:33:37.066734 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:37.066695 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86bc23cf-ct5rg"] Oct 30 13:33:37.358362 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:33:37.358241 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:33:37.664154 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:37.664038 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d50635b7-b44f-48ff-aea5-dad300e22705" path="/var/lib/kubelet/pods/d50635b7-b44f-48ff-aea5-dad300e22705/volumes" Oct 30 13:33:38.501046 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:38.501012 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn"] Oct 30 13:33:38.506760 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:38.506738 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:33:38.524562 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:38.524518 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-78543b-openshift-default-dockercfg-rdc47\"" Oct 30 13:33:38.546622 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:38.546595 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn"] Oct 30 13:33:38.573903 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:38.573873 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-workload-socket\") pod \"e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:33:38.574111 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:38.574084 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-istio-podinfo\") pod \"e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:33:38.574230 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:38.574218 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-workload-certs\") pod \"e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:33:38.574317 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:38.574246 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjk9z\" (UniqueName: \"kubernetes.io/projected/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-kube-api-access-hjk9z\") pod \"e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:33:38.574463 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:38.574438 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-istio-token\") pod \"e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:33:38.574595 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:38.574569 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-istio-envoy\") pod \"e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:33:38.574708 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:38.574633 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-credential-socket\") pod \"e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:33:38.574927 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:38.574903 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-istio-data\") pod \"e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:33:38.575030 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:38.575012 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-istiod-ca-cert\") pod \"e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:33:38.717346 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:38.678312 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-istio-token\") pod \"e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:33:38.736958 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:38.736481 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-istio-envoy\") pod \"e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:33:38.737286 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:38.737242 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-credential-socket\") pod \"e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:33:38.737512 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:38.737492 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-istio-data\") pod \"e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:33:38.737682 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:38.737668 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-istiod-ca-cert\") pod \"e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:33:38.737909 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:38.737888 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-workload-socket\") pod \"e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:33:38.758334 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:38.738668 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-istio-podinfo\") pod \"e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:33:38.758655 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:38.758628 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-workload-certs\") pod \"e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:33:38.758821 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:38.758804 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjk9z\" (UniqueName: \"kubernetes.io/projected/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-kube-api-access-hjk9z\") pod \"e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:33:38.759607 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:38.739638 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-credential-socket\") pod \"e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:33:38.767296 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:38.756656 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-istio-data\") pod \"e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:33:38.767787 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:38.767759 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-workload-certs\") pod \"e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:33:38.776290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:38.740308 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-istiod-ca-cert\") pod \"e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:33:38.776467 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:38.740555 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-workload-socket\") pod \"e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:33:38.794304 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:38.787735 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-istio-envoy\") pod \"e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:33:38.812295 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:38.808623 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-istio-podinfo\") pod \"e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:33:38.821479 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:38.821447 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-istio-token\") pod \"e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:33:38.828397 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:38.828367 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjk9z\" (UniqueName: \"kubernetes.io/projected/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-kube-api-access-hjk9z\") pod \"e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:33:38.840635 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:38.840597 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:33:39.684332 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:33:39.681489 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb446da63_fda1_43e4_91b0_9a2d2a77b1a1.slice/crio-312b7f83ff9d992297cbf7a075078661e5b575456f437ec507ceceafe92530ea WatchSource:0}: Error finding container 312b7f83ff9d992297cbf7a075078661e5b575456f437ec507ceceafe92530ea: Status 404 returned error can't find the container with id 312b7f83ff9d992297cbf7a075078661e5b575456f437ec507ceceafe92530ea Oct 30 13:33:39.692564 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:39.691383 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:33:39.692829 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:39.692799 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:33:39.692974 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:39.692958 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:33:39.721380 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:39.721343 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:33:39.814423 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:39.813925 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn"] Oct 30 13:33:40.048596 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:40.048555 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" event={"ID":"b446da63-fda1-43e4-91b0-9a2d2a77b1a1","Type":"ContainerStarted","Data":"ceefc64030e945fa07b7d162f07b292ce681a38af7d32cce715606702a3af29d"} Oct 30 13:33:40.048866 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:40.048838 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" event={"ID":"b446da63-fda1-43e4-91b0-9a2d2a77b1a1","Type":"ContainerStarted","Data":"312b7f83ff9d992297cbf7a075078661e5b575456f437ec507ceceafe92530ea"} Oct 30 13:33:40.109522 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:40.109412 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" podStartSLOduration=2.109392616 podStartE2EDuration="2.109392616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:40.106647748 +0000 UTC m=+2918.710146195" watchObservedRunningTime="2026-10-30 13:33:40.109392616 +0000 UTC m=+2918.712891063" Oct 30 13:33:40.857289 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:40.848971 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:33:40.895289 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:40.891470 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:33:41.215305 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:41.204084 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:33:41.223292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:41.221521 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:33:41.948322 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:41.923659 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" podUID="9577f571-30cd-4043-a3e4-72bf95837552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:33:42.083294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:42.081358 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" podUID="7ad1989a-c73f-4647-9e50-855605bdb08e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:33:42.198461 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:42.197396 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" podUID="b0d31d29-93c6-4992-b3b7-1bb491b382fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:33:42.237292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:42.224820 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" podUID="d345fdd5-407b-4162-a2cd-64a40068cb3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:33:42.237579 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:42.237533 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" podUID="5a504237-e5a3-491f-8f84-0419cf502f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:33:42.389288 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:42.373113 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" podUID="f3b342db-9a01-46ad-b476-bfc912d9ce1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:33:42.714287 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:42.702328 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" podUID="65799c14-8de4-4b4a-90df-7cd257e1e581" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:33:42.788296 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:42.787409 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp" podUID="c5601047-51b4-4310-9289-580b8891746e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:33:42.926345 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:42.925354 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" podUID="9577f571-30cd-4043-a3e4-72bf95837552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:33:42.926345 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:42.926127 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:33:42.963118 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:42.963084 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:33:42.999525 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:42.993363 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" podUID="089cb620-913d-47c7-b8f7-9b2298c194bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 13:33:43.162622 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:43.156535 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" podUID="0c5a9f00-6d5b-4860-a210-2fec24afb322" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:33:43.169383 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:43.169342 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" podUID="b1e86eea-4824-4c50-8ad3-2a15c3365c3d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:33:43.272976 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:43.272832 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" podUID="5a504237-e5a3-491f-8f84-0419cf502f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 13:33:43.295614 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:43.295569 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" podUID="d345fdd5-407b-4162-a2cd-64a40068cb3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 13:33:43.317083 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:43.297708 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:33:43.317823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:43.317798 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 13:33:43.323328 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:43.304663 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" podUID="b0d31d29-93c6-4992-b3b7-1bb491b382fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": context deadline exceeded" Oct 30 13:33:43.323830 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:43.323813 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" Oct 30 13:33:43.355583 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:43.355551 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 13:33:43.355871 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:43.355853 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:33:43.472011 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:43.471979 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" Oct 30 13:33:47.060728 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:47.060690 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-475f32cc-smnnr"] Oct 30 13:33:47.077124 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:47.077092 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-475f32cc-smnnr"] Oct 30 13:33:47.135480 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:47.135407 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r" podUID="f5f88311-b26d-4433-804e-5a2f05180694" containerName="main" containerID="cri-o://5afffceba04b175616d91d7137f3617f84b1fc774f13329650b5573b7cdbee0d" gracePeriod=45 Oct 30 13:33:47.535014 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:47.534981 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09283563-d7cc-43f6-b34a-e683eae58317" path="/var/lib/kubelet/pods/09283563-d7cc-43f6-b34a-e683eae58317/volumes" Oct 30 13:33:47.548457 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:47.548305 4931 generic.go:356] "Generic (PLEG): container finished" podID="f5f88311-b26d-4433-804e-5a2f05180694" containerID="5afffceba04b175616d91d7137f3617f84b1fc774f13329650b5573b7cdbee0d" exitCode=0 Oct 30 13:33:47.548457 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:47.548366 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r" event={"ID":"f5f88311-b26d-4433-804e-5a2f05180694","Type":"ContainerDied","Data":"5afffceba04b175616d91d7137f3617f84b1fc774f13329650b5573b7cdbee0d"} Oct 30 13:33:47.548457 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:47.548417 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r" event={"ID":"f5f88311-b26d-4433-804e-5a2f05180694","Type":"ContainerDied","Data":"0e90a62babe87371c64017bdcba8d74ad00ddc3f4c22ef1c9ab9dee3ff89a4e4"} Oct 30 13:33:47.548457 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:47.548432 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e90a62babe87371c64017bdcba8d74ad00ddc3f4c22ef1c9ab9dee3ff89a4e4" Oct 30 13:33:47.559158 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:47.559134 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r" Oct 30 13:33:47.649165 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:47.649068 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5f88311-b26d-4433-804e-5a2f05180694-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5f88311-b26d-4433-804e-5a2f05180694-dshm\") pod \"f5f88311-b26d-4433-804e-5a2f05180694\" (UID: \"f5f88311-b26d-4433-804e-5a2f05180694\") " Oct 30 13:33:47.649401 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:47.649282 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5f88311-b26d-4433-804e-5a2f05180694-kube-api-access-2vqbq\" (UniqueName: \"kubernetes.io/projected/f5f88311-b26d-4433-804e-5a2f05180694-kube-api-access-2vqbq\") pod \"f5f88311-b26d-4433-804e-5a2f05180694\" (UID: \"f5f88311-b26d-4433-804e-5a2f05180694\") " Oct 30 13:33:47.649527 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:47.649504 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5f88311-b26d-4433-804e-5a2f05180694-tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f88311-b26d-4433-804e-5a2f05180694-tls-certs\") pod \"f5f88311-b26d-4433-804e-5a2f05180694\" (UID: \"f5f88311-b26d-4433-804e-5a2f05180694\") " Oct 30 13:33:47.649643 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:47.649624 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5f88311-b26d-4433-804e-5a2f05180694-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5f88311-b26d-4433-804e-5a2f05180694-tmp-dir\") pod \"f5f88311-b26d-4433-804e-5a2f05180694\" (UID: \"f5f88311-b26d-4433-804e-5a2f05180694\") " Oct 30 13:33:47.649734 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:47.649715 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5f88311-b26d-4433-804e-5a2f05180694-home\" (UniqueName: \"kubernetes.io/empty-dir/f5f88311-b26d-4433-804e-5a2f05180694-home\") pod \"f5f88311-b26d-4433-804e-5a2f05180694\" (UID: \"f5f88311-b26d-4433-804e-5a2f05180694\") " Oct 30 13:33:47.649816 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:47.649797 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5f88311-b26d-4433-804e-5a2f05180694-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f88311-b26d-4433-804e-5a2f05180694-model-cache\") pod \"f5f88311-b26d-4433-804e-5a2f05180694\" (UID: \"f5f88311-b26d-4433-804e-5a2f05180694\") " Oct 30 13:33:47.651167 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:47.651121 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f88311-b26d-4433-804e-5a2f05180694-model-cache" pod "f5f88311-b26d-4433-804e-5a2f05180694" (UID: "f5f88311-b26d-4433-804e-5a2f05180694"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:33:47.651445 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:47.651348 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f88311-b26d-4433-804e-5a2f05180694-tmp-dir" pod "f5f88311-b26d-4433-804e-5a2f05180694" (UID: "f5f88311-b26d-4433-804e-5a2f05180694"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:33:47.651588 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:47.651566 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f88311-b26d-4433-804e-5a2f05180694-home" pod "f5f88311-b26d-4433-804e-5a2f05180694" (UID: "f5f88311-b26d-4433-804e-5a2f05180694"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:33:47.654831 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:47.654720 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f88311-b26d-4433-804e-5a2f05180694-dshm" pod "f5f88311-b26d-4433-804e-5a2f05180694" (UID: "f5f88311-b26d-4433-804e-5a2f05180694"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:33:47.655732 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:47.655689 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5f88311-b26d-4433-804e-5a2f05180694-kube-api-access-2vqbq" pod "f5f88311-b26d-4433-804e-5a2f05180694" (UID: "f5f88311-b26d-4433-804e-5a2f05180694"). InnerVolumeSpecName "kube-api-access-2vqbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:33:47.656366 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:47.656339 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5f88311-b26d-4433-804e-5a2f05180694-tls-certs" pod "f5f88311-b26d-4433-804e-5a2f05180694" (UID: "f5f88311-b26d-4433-804e-5a2f05180694"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:33:47.752850 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:47.752809 4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5f88311-b26d-4433-804e-5a2f05180694-dshm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:33:47.752850 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:47.752842 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vqbq\" (UniqueName: \"kubernetes.io/projected/f5f88311-b26d-4433-804e-5a2f05180694-kube-api-access-2vqbq\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:33:47.753041 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:47.752857 4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f88311-b26d-4433-804e-5a2f05180694-tls-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:33:47.753041 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:47.752869 4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5f88311-b26d-4433-804e-5a2f05180694-tmp-dir\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:33:47.753041 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:47.752882 4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5f88311-b26d-4433-804e-5a2f05180694-home\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:33:47.753041 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:47.752895 4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f88311-b26d-4433-804e-5a2f05180694-model-cache\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:33:48.529236 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:48.529195 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r" podUID="f5f88311-b26d-4433-804e-5a2f05180694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.186:8000/ready\": context deadline exceeded" Oct 30 13:33:48.556935 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:48.556905 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r" Oct 30 13:33:48.630487 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:48.630452 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r"] Oct 30 13:33:48.638806 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:48.638775 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-acd125/rate-test-model-acd125-kserve-56c8f5f9cc-jlq2r"] Oct 30 13:33:49.516612 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:49.516574 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5f88311-b26d-4433-804e-5a2f05180694" path="/var/lib/kubelet/pods/f5f88311-b26d-4433-804e-5a2f05180694/volumes" Oct 30 13:33:52.630330 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:52.628114 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p"] Oct 30 13:33:52.630330 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:52.629188 4931 cpu_manager.go:414] "containerMap: removing container" podUID="f5f88311-b26d-4433-804e-5a2f05180694" containerName="main" Oct 30 13:33:52.630330 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:52.629206 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5f88311-b26d-4433-804e-5a2f05180694" containerName="main" Oct 30 13:33:52.630330 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:52.629370 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5f88311-b26d-4433-804e-5a2f05180694" containerName="main" Oct 30 13:33:52.636599 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:52.636570 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p" Oct 30 13:33:52.636983 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:52.636960 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p"] Oct 30 13:33:52.691160 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:52.691121 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3e549a03-6e0a-4068-81f1-8747bc5570a0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-78h7p\" (UID: \"3e549a03-6e0a-4068-81f1-8747bc5570a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p" Oct 30 13:33:52.691361 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:52.691313 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e549a03-6e0a-4068-81f1-8747bc5570a0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-78h7p\" (UID: \"3e549a03-6e0a-4068-81f1-8747bc5570a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p" Oct 30 13:33:52.691441 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:52.691390 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3e549a03-6e0a-4068-81f1-8747bc5570a0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-78h7p\" (UID: \"3e549a03-6e0a-4068-81f1-8747bc5570a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p" Oct 30 13:33:52.691441 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:52.691430 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3e549a03-6e0a-4068-81f1-8747bc5570a0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-78h7p\" (UID: \"3e549a03-6e0a-4068-81f1-8747bc5570a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p" Oct 30 13:33:52.691563 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:52.691521 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e549a03-6e0a-4068-81f1-8747bc5570a0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-78h7p\" (UID: \"3e549a03-6e0a-4068-81f1-8747bc5570a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p" Oct 30 13:33:52.691563 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:52.691543 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd2m4\" (UniqueName: \"kubernetes.io/projected/3e549a03-6e0a-4068-81f1-8747bc5570a0-kube-api-access-nd2m4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-78h7p\" (UID: \"3e549a03-6e0a-4068-81f1-8747bc5570a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p" Oct 30 13:33:52.691675 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:52.691663 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3e549a03-6e0a-4068-81f1-8747bc5570a0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-78h7p\" (UID: \"3e549a03-6e0a-4068-81f1-8747bc5570a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p" Oct 30 13:33:52.793394 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:52.793356 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3e549a03-6e0a-4068-81f1-8747bc5570a0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-78h7p\" (UID: \"3e549a03-6e0a-4068-81f1-8747bc5570a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p" Oct 30 13:33:52.793593 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:52.793458 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3e549a03-6e0a-4068-81f1-8747bc5570a0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-78h7p\" (UID: \"3e549a03-6e0a-4068-81f1-8747bc5570a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p" Oct 30 13:33:52.793670 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:52.793617 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e549a03-6e0a-4068-81f1-8747bc5570a0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-78h7p\" (UID: \"3e549a03-6e0a-4068-81f1-8747bc5570a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p" Oct 30 13:33:52.793731 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:52.793678 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3e549a03-6e0a-4068-81f1-8747bc5570a0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-78h7p\" (UID: \"3e549a03-6e0a-4068-81f1-8747bc5570a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p" Oct 30 13:33:52.793731 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:52.793699 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3e549a03-6e0a-4068-81f1-8747bc5570a0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-78h7p\" (UID: \"3e549a03-6e0a-4068-81f1-8747bc5570a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p" Oct 30 13:33:52.793847 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:52.793761 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e549a03-6e0a-4068-81f1-8747bc5570a0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-78h7p\" (UID: \"3e549a03-6e0a-4068-81f1-8747bc5570a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p" Oct 30 13:33:52.793847 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:52.793784 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd2m4\" (UniqueName: \"kubernetes.io/projected/3e549a03-6e0a-4068-81f1-8747bc5570a0-kube-api-access-nd2m4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-78h7p\" (UID: \"3e549a03-6e0a-4068-81f1-8747bc5570a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p" Oct 30 13:33:52.794737 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:52.794702 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3e549a03-6e0a-4068-81f1-8747bc5570a0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-78h7p\" (UID: \"3e549a03-6e0a-4068-81f1-8747bc5570a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p" Oct 30 13:33:52.796437 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:52.796401 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3e549a03-6e0a-4068-81f1-8747bc5570a0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-78h7p\" (UID: \"3e549a03-6e0a-4068-81f1-8747bc5570a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p" Oct 30 13:33:52.796578 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:52.796405 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e549a03-6e0a-4068-81f1-8747bc5570a0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-78h7p\" (UID: \"3e549a03-6e0a-4068-81f1-8747bc5570a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p" Oct 30 13:33:52.796578 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:52.796461 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3e549a03-6e0a-4068-81f1-8747bc5570a0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-78h7p\" (UID: \"3e549a03-6e0a-4068-81f1-8747bc5570a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p" Oct 30 13:33:52.796578 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:52.796538 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3e549a03-6e0a-4068-81f1-8747bc5570a0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-78h7p\" (UID: \"3e549a03-6e0a-4068-81f1-8747bc5570a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p" Oct 30 13:33:52.819379 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:52.819352 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e549a03-6e0a-4068-81f1-8747bc5570a0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-78h7p\" (UID: \"3e549a03-6e0a-4068-81f1-8747bc5570a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p" Oct 30 13:33:52.820065 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:52.820040 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd2m4\" (UniqueName: \"kubernetes.io/projected/3e549a03-6e0a-4068-81f1-8747bc5570a0-kube-api-access-nd2m4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-78h7p\" (UID: \"3e549a03-6e0a-4068-81f1-8747bc5570a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p" Oct 30 13:33:52.954407 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:52.954374 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p" Oct 30 13:33:53.305684 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:33:53.305650 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e549a03_6e0a_4068_81f1_8747bc5570a0.slice/crio-96160c80f36dc9f0fd96b58ef4bdb995e23af732c079612fe7b12366457ea5ea WatchSource:0}: Error finding container 96160c80f36dc9f0fd96b58ef4bdb995e23af732c079612fe7b12366457ea5ea: Status 404 returned error can't find the container with id 96160c80f36dc9f0fd96b58ef4bdb995e23af732c079612fe7b12366457ea5ea Oct 30 13:33:53.308260 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:53.308236 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p"] Oct 30 13:33:53.309159 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:53.309125 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:33:53.309224 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:53.309190 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:33:53.655858 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:53.655752 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p" event={"ID":"3e549a03-6e0a-4068-81f1-8747bc5570a0","Type":"ContainerStarted","Data":"b3f3073914fd8bd72a3f09fa179292d06caf2a19199b7bb14beb70ec4152b136"} Oct 30 13:33:53.655858 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:53.655819 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p" event={"ID":"3e549a03-6e0a-4068-81f1-8747bc5570a0","Type":"ContainerStarted","Data":"96160c80f36dc9f0fd96b58ef4bdb995e23af732c079612fe7b12366457ea5ea"} Oct 30 13:33:53.656321 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:53.655870 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p" Oct 30 13:33:53.723636 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:53.723579 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p" podStartSLOduration=1.723564818 podStartE2EDuration="1.723564818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:53.7212078 +0000 UTC m=+2932.324706260" watchObservedRunningTime="2026-10-30 13:33:53.723564818 +0000 UTC m=+2932.327063264" Oct 30 13:33:54.668796 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:54.668757 4931 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-78h7p container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:33:54.669431 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:54.668813 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p" podUID="3e549a03-6e0a-4068-81f1-8747bc5570a0" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:33:57.667947 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:33:57.667909 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p" Oct 30 13:34:03.600956 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:34:03.600919 4931 scope.go:122] "RemoveContainer" containerID="883c0d16edc9c250e050ae9f56756f71bf6948b41eacadbc928690a967790464" Oct 30 13:34:03.613242 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:34:03.613218 4931 scope.go:122] "RemoveContainer" containerID="11ca4a3af61346d91d71de9c629fcb0ca70c8fa8b50bff3898a1d1f842b2b977" Oct 30 13:34:03.624579 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:34:03.624489 4931 scope.go:122] "RemoveContainer" containerID="23b0cbcb6bd270adcf5161327e93d8d1cca3546f56b0752356f65e228e3128e9" Oct 30 13:34:07.354462 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:34:07.354423 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:34:13.991593 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:34:13.976475 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" podUID="9577f571-30cd-4043-a3e4-72bf95837552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:34:31.043094 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:34:31.043057 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1efa867-tms6c"] Oct 30 13:34:31.054835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:34:31.054807 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1efa867-tms6c"] Oct 30 13:34:31.515172 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:34:31.515134 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2be096ed-3a80-4449-8ea1-3281d1fa5ba5" path="/var/lib/kubelet/pods/2be096ed-3a80-4449-8ea1-3281d1fa5ba5/volumes" Oct 30 13:34:37.354738 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:34:37.354696 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:34:42.048865 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:34:42.048828 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d9f4c53-htbbm"] Oct 30 13:34:42.061795 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:34:42.061765 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d9f4c53-htbbm"] Oct 30 13:34:43.515558 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:34:43.515522 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3324d5df-e2ef-46ab-89b2-3b64126732b9" path="/var/lib/kubelet/pods/3324d5df-e2ef-46ab-89b2-3b64126732b9/volumes" Oct 30 13:34:49.151459 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:34:49.151409 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:34:52.048424 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:34:52.048387 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc79a401-n6895"] Oct 30 13:34:52.053200 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:34:52.053169 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc79a401-n6895"] Oct 30 13:34:53.517950 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:34:53.517914 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f22bdb08-5157-454d-a5a5-b94f96d135fb" path="/var/lib/kubelet/pods/f22bdb08-5157-454d-a5a5-b94f96d135fb/volumes" Oct 30 13:35:02.608485 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:02.608457 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/2.log" Oct 30 13:35:02.611463 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:02.611438 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/2.log" Oct 30 13:35:03.667109 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:03.667074 4931 scope.go:122] "RemoveContainer" containerID="fb543fd7dae3a86556c1747536d0658860722d2b99dcebbac77ec65d691b2434" Oct 30 13:35:03.678383 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:03.678364 4931 scope.go:122] "RemoveContainer" containerID="d1cb79df2f61e26246f0d70a95b39f3ce7c8a663c401ad801480f6fd53ce8f33" Oct 30 13:35:03.688388 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:03.688368 4931 scope.go:122] "RemoveContainer" containerID="d5a34cadd6bc97fc8278b5a8f2c7f6c454577ecaf794b1b45d565973a6111854" Oct 30 13:35:03.698574 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:03.698546 4931 scope.go:122] "RemoveContainer" containerID="7452905f511b68dc31af6a9e8efa75b8e87cbabbd5380daeb75144c6b9f1dfa1" Oct 30 13:35:03.708340 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:03.708315 4931 scope.go:122] "RemoveContainer" containerID="625f65074c752fb757e27289d4a460b75cfebba9149f5d8e9801726a7ce83de6" Oct 30 13:35:03.717564 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:03.717542 4931 scope.go:122] "RemoveContainer" containerID="60b23916487596c5730fc594672c71185b3755a7f8ebe49ce6ec673a4ac92723" Oct 30 13:35:05.049066 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:05.049029 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5420598-fmxqg"] Oct 30 13:35:05.062084 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:05.062056 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5420598-fmxqg"] Oct 30 13:35:05.514862 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:05.514826 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0716cec-3e99-4ef4-ad57-61dc8bfc9c75" path="/var/lib/kubelet/pods/b0716cec-3e99-4ef4-ad57-61dc8bfc9c75/volumes" Oct 30 13:35:07.354943 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:35:07.354897 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:35:14.806239 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:14.806043 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj"] Oct 30 13:35:15.357523 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:15.357484 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp"] Oct 30 13:35:26.418193 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:26.418153 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts"] Oct 30 13:35:27.245445 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:27.245392 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:35:27.245667 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:27.245472 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" podUID="b446da63-fda1-43e4-91b0-9a2d2a77b1a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:35:29.844484 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:29.844447 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj" podUID="6972ffa9-c402-415f-8166-7db0e5d8097c" containerName="main" containerID="cri-o://7a0ea744c9a0e49ae5ed367056c75215c405e9d57ff28b51f108a6b32e5bcb43" gracePeriod=45 Oct 30 13:35:29.994034 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:29.993997 4931 generic.go:356] "Generic (PLEG): container finished" podID="6972ffa9-c402-415f-8166-7db0e5d8097c" containerID="7a0ea744c9a0e49ae5ed367056c75215c405e9d57ff28b51f108a6b32e5bcb43" exitCode=0 Oct 30 13:35:29.994251 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:29.994154 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj" event={"ID":"6972ffa9-c402-415f-8166-7db0e5d8097c","Type":"ContainerDied","Data":"7a0ea744c9a0e49ae5ed367056c75215c405e9d57ff28b51f108a6b32e5bcb43"} Oct 30 13:35:29.995973 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:29.995949 4931 generic.go:356] "Generic (PLEG): container finished" podID="b446da63-fda1-43e4-91b0-9a2d2a77b1a1" containerID="ceefc64030e945fa07b7d162f07b292ce681a38af7d32cce715606702a3af29d" exitCode=0 Oct 30 13:35:29.996096 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:29.996019 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" event={"ID":"b446da63-fda1-43e4-91b0-9a2d2a77b1a1","Type":"ContainerDied","Data":"ceefc64030e945fa07b7d162f07b292ce681a38af7d32cce715606702a3af29d"} Oct 30 13:35:29.996568 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:29.996552 4931 scope.go:122] "RemoveContainer" containerID="ceefc64030e945fa07b7d162f07b292ce681a38af7d32cce715606702a3af29d" Oct 30 13:35:29.999159 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:29.999122 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:35:29.999253 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:29.999223 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:35:29.999328 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:29.999290 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:35:30.323252 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.323223 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj" Oct 30 13:35:30.354713 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.354628 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6972ffa9-c402-415f-8166-7db0e5d8097c-tls-certs\" (UniqueName: \"kubernetes.io/secret/6972ffa9-c402-415f-8166-7db0e5d8097c-tls-certs\") pod \"6972ffa9-c402-415f-8166-7db0e5d8097c\" (UID: \"6972ffa9-c402-415f-8166-7db0e5d8097c\") " Oct 30 13:35:30.354893 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.354721 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6972ffa9-c402-415f-8166-7db0e5d8097c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6972ffa9-c402-415f-8166-7db0e5d8097c-model-cache\") pod \"6972ffa9-c402-415f-8166-7db0e5d8097c\" (UID: \"6972ffa9-c402-415f-8166-7db0e5d8097c\") " Oct 30 13:35:30.354893 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.354859 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6972ffa9-c402-415f-8166-7db0e5d8097c-kube-api-access-jm2wr\" (UniqueName: \"kubernetes.io/projected/6972ffa9-c402-415f-8166-7db0e5d8097c-kube-api-access-jm2wr\") pod \"6972ffa9-c402-415f-8166-7db0e5d8097c\" (UID: \"6972ffa9-c402-415f-8166-7db0e5d8097c\") " Oct 30 13:35:30.355035 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.354927 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6972ffa9-c402-415f-8166-7db0e5d8097c-home\" (UniqueName: \"kubernetes.io/empty-dir/6972ffa9-c402-415f-8166-7db0e5d8097c-home\") pod \"6972ffa9-c402-415f-8166-7db0e5d8097c\" (UID: \"6972ffa9-c402-415f-8166-7db0e5d8097c\") " Oct 30 13:35:30.355035 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.354950 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6972ffa9-c402-415f-8166-7db0e5d8097c-model-cache" pod "6972ffa9-c402-415f-8166-7db0e5d8097c" (UID: "6972ffa9-c402-415f-8166-7db0e5d8097c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:30.355035 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.355016 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6972ffa9-c402-415f-8166-7db0e5d8097c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6972ffa9-c402-415f-8166-7db0e5d8097c-dshm\") pod \"6972ffa9-c402-415f-8166-7db0e5d8097c\" (UID: \"6972ffa9-c402-415f-8166-7db0e5d8097c\") " Oct 30 13:35:30.355195 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.355070 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6972ffa9-c402-415f-8166-7db0e5d8097c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6972ffa9-c402-415f-8166-7db0e5d8097c-tmp-dir\") pod \"6972ffa9-c402-415f-8166-7db0e5d8097c\" (UID: \"6972ffa9-c402-415f-8166-7db0e5d8097c\") " Oct 30 13:35:30.355278 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.355242 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6972ffa9-c402-415f-8166-7db0e5d8097c-home" pod "6972ffa9-c402-415f-8166-7db0e5d8097c" (UID: "6972ffa9-c402-415f-8166-7db0e5d8097c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:30.355539 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.355499 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6972ffa9-c402-415f-8166-7db0e5d8097c-tmp-dir" pod "6972ffa9-c402-415f-8166-7db0e5d8097c" (UID: "6972ffa9-c402-415f-8166-7db0e5d8097c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:30.355832 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.355788 4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6972ffa9-c402-415f-8166-7db0e5d8097c-home\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:35:30.355832 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.355815 4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6972ffa9-c402-415f-8166-7db0e5d8097c-tmp-dir\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:35:30.355832 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.355830 4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6972ffa9-c402-415f-8166-7db0e5d8097c-model-cache\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:35:30.357754 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.357730 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6972ffa9-c402-415f-8166-7db0e5d8097c-tls-certs" pod "6972ffa9-c402-415f-8166-7db0e5d8097c" (UID: "6972ffa9-c402-415f-8166-7db0e5d8097c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:35:30.357914 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.357792 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6972ffa9-c402-415f-8166-7db0e5d8097c-kube-api-access-jm2wr" pod "6972ffa9-c402-415f-8166-7db0e5d8097c" (UID: "6972ffa9-c402-415f-8166-7db0e5d8097c"). InnerVolumeSpecName "kube-api-access-jm2wr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:35:30.358200 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.358183 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6972ffa9-c402-415f-8166-7db0e5d8097c-dshm" pod "6972ffa9-c402-415f-8166-7db0e5d8097c" (UID: "6972ffa9-c402-415f-8166-7db0e5d8097c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:30.386397 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.386326 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp" podUID="c5601047-51b4-4310-9289-580b8891746e" containerName="main" containerID="cri-o://2699bf11018dbdbd981a905373cba75d3029c0b22511e87e414137719688c5c4" gracePeriod=45 Oct 30 13:35:30.457589 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.457542 4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6972ffa9-c402-415f-8166-7db0e5d8097c-tls-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:35:30.457589 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.457575 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm2wr\" (UniqueName: \"kubernetes.io/projected/6972ffa9-c402-415f-8166-7db0e5d8097c-kube-api-access-jm2wr\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:35:30.457589 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.457592 4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6972ffa9-c402-415f-8166-7db0e5d8097c-dshm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:35:30.680230 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.680200 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp" Oct 30 13:35:30.762344 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.762246 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5601047-51b4-4310-9289-580b8891746e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5601047-51b4-4310-9289-580b8891746e-dshm\") pod \"c5601047-51b4-4310-9289-580b8891746e\" (UID: \"c5601047-51b4-4310-9289-580b8891746e\") " Oct 30 13:35:30.762539 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.762435 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5601047-51b4-4310-9289-580b8891746e-kube-api-access-k2zhv\" (UniqueName: \"kubernetes.io/projected/c5601047-51b4-4310-9289-580b8891746e-kube-api-access-k2zhv\") pod \"c5601047-51b4-4310-9289-580b8891746e\" (UID: \"c5601047-51b4-4310-9289-580b8891746e\") " Oct 30 13:35:30.762539 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.762490 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5601047-51b4-4310-9289-580b8891746e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5601047-51b4-4310-9289-580b8891746e-tmp-dir\") pod \"c5601047-51b4-4310-9289-580b8891746e\" (UID: \"c5601047-51b4-4310-9289-580b8891746e\") " Oct 30 13:35:30.762705 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.762675 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5601047-51b4-4310-9289-580b8891746e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5601047-51b4-4310-9289-580b8891746e-model-cache\") pod \"c5601047-51b4-4310-9289-580b8891746e\" (UID: \"c5601047-51b4-4310-9289-580b8891746e\") " Oct 30 13:35:30.762826 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.762783 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5601047-51b4-4310-9289-580b8891746e-home\" (UniqueName: \"kubernetes.io/empty-dir/c5601047-51b4-4310-9289-580b8891746e-home\") pod \"c5601047-51b4-4310-9289-580b8891746e\" (UID: \"c5601047-51b4-4310-9289-580b8891746e\") " Oct 30 13:35:30.762917 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.762827 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5601047-51b4-4310-9289-580b8891746e-tls-certs\" (UniqueName: \"kubernetes.io/secret/c5601047-51b4-4310-9289-580b8891746e-tls-certs\") pod \"c5601047-51b4-4310-9289-580b8891746e\" (UID: \"c5601047-51b4-4310-9289-580b8891746e\") " Oct 30 13:35:30.762917 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.762896 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5601047-51b4-4310-9289-580b8891746e-tmp-dir" pod "c5601047-51b4-4310-9289-580b8891746e" (UID: "c5601047-51b4-4310-9289-580b8891746e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:30.763165 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.763121 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5601047-51b4-4310-9289-580b8891746e-model-cache" pod "c5601047-51b4-4310-9289-580b8891746e" (UID: "c5601047-51b4-4310-9289-580b8891746e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:30.763646 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.763622 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5601047-51b4-4310-9289-580b8891746e-home" pod "c5601047-51b4-4310-9289-580b8891746e" (UID: "c5601047-51b4-4310-9289-580b8891746e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:30.763997 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.763701 4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5601047-51b4-4310-9289-580b8891746e-tmp-dir\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:35:30.764198 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.764178 4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5601047-51b4-4310-9289-580b8891746e-model-cache\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:35:30.766198 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.766164 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5601047-51b4-4310-9289-580b8891746e-dshm" pod "c5601047-51b4-4310-9289-580b8891746e" (UID: "c5601047-51b4-4310-9289-580b8891746e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:30.766367 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.766247 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5601047-51b4-4310-9289-580b8891746e-kube-api-access-k2zhv" pod "c5601047-51b4-4310-9289-580b8891746e" (UID: "c5601047-51b4-4310-9289-580b8891746e"). InnerVolumeSpecName "kube-api-access-k2zhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:35:30.766367 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.766262 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5601047-51b4-4310-9289-580b8891746e-tls-certs" pod "c5601047-51b4-4310-9289-580b8891746e" (UID: "c5601047-51b4-4310-9289-580b8891746e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:35:30.866522 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.866490 4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5601047-51b4-4310-9289-580b8891746e-home\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:35:30.866863 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.866523 4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5601047-51b4-4310-9289-580b8891746e-tls-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:35:30.866863 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.866538 4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5601047-51b4-4310-9289-580b8891746e-dshm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:35:30.866863 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:30.866551 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2zhv\" (UniqueName: \"kubernetes.io/projected/c5601047-51b4-4310-9289-580b8891746e-kube-api-access-k2zhv\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:35:31.006526 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:31.006488 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" event={"ID":"b446da63-fda1-43e4-91b0-9a2d2a77b1a1","Type":"ContainerStarted","Data":"f058bc5026bf2957d921ed6b72ac80ae5973e89a4e3f4473db01c8d4d0954853"} Oct 30 13:35:31.008902 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:31.008879 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj" Oct 30 13:35:31.009534 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:31.009484 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj" event={"ID":"6972ffa9-c402-415f-8166-7db0e5d8097c","Type":"ContainerDied","Data":"62a32e63fbf4fbb549faf84bd0ad6580f98f6123584c64015e7afc7fac569216"} Oct 30 13:35:31.009661 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:31.009544 4931 scope.go:122] "RemoveContainer" containerID="7a0ea744c9a0e49ae5ed367056c75215c405e9d57ff28b51f108a6b32e5bcb43" Oct 30 13:35:31.012695 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:31.012669 4931 generic.go:356] "Generic (PLEG): container finished" podID="c5601047-51b4-4310-9289-580b8891746e" containerID="2699bf11018dbdbd981a905373cba75d3029c0b22511e87e414137719688c5c4" exitCode=0 Oct 30 13:35:31.012835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:31.012708 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp" Oct 30 13:35:31.012835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:31.012727 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp" event={"ID":"c5601047-51b4-4310-9289-580b8891746e","Type":"ContainerDied","Data":"2699bf11018dbdbd981a905373cba75d3029c0b22511e87e414137719688c5c4"} Oct 30 13:35:31.012835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:31.012754 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp" event={"ID":"c5601047-51b4-4310-9289-580b8891746e","Type":"ContainerDied","Data":"ea490111ceadc59c5412a40fd9e29d1216c75095f3b4e724969d7fdd81accbfd"} Oct 30 13:35:31.023566 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:31.023542 4931 scope.go:122] "RemoveContainer" containerID="2699bf11018dbdbd981a905373cba75d3029c0b22511e87e414137719688c5c4" Oct 30 13:35:31.034014 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:31.033994 4931 scope.go:122] "RemoveContainer" containerID="2699bf11018dbdbd981a905373cba75d3029c0b22511e87e414137719688c5c4" Oct 30 13:35:31.034308 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:35:31.034245 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2699bf11018dbdbd981a905373cba75d3029c0b22511e87e414137719688c5c4\": container with ID starting with 2699bf11018dbdbd981a905373cba75d3029c0b22511e87e414137719688c5c4 not found: ID does not exist" containerID="2699bf11018dbdbd981a905373cba75d3029c0b22511e87e414137719688c5c4" Oct 30 13:35:31.034308 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:31.034289 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2699bf11018dbdbd981a905373cba75d3029c0b22511e87e414137719688c5c4"} err="failed to get container status \"2699bf11018dbdbd981a905373cba75d3029c0b22511e87e414137719688c5c4\": rpc error: code = NotFound desc = could not find container \"2699bf11018dbdbd981a905373cba75d3029c0b22511e87e414137719688c5c4\": container with ID starting with 2699bf11018dbdbd981a905373cba75d3029c0b22511e87e414137719688c5c4 not found: ID does not exist" Oct 30 13:35:31.105483 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:31.105449 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj"] Oct 30 13:35:31.110371 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:31.110342 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj"] Oct 30 13:35:31.157174 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:31.157102 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp"] Oct 30 13:35:31.176295 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:31.173514 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp"] Oct 30 13:35:31.258350 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:31.254368 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-51288b/rate-test-model-51288b-kserve-5b466c777c-knhfj" podUID="6972ffa9-c402-415f-8166-7db0e5d8097c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:35:31.515554 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:31.515476 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6972ffa9-c402-415f-8166-7db0e5d8097c" path="/var/lib/kubelet/pods/6972ffa9-c402-415f-8166-7db0e5d8097c/volumes" Oct 30 13:35:31.516054 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:31.516038 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5601047-51b4-4310-9289-580b8891746e" path="/var/lib/kubelet/pods/c5601047-51b4-4310-9289-580b8891746e/volumes" Oct 30 13:35:31.619752 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:31.619706 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-1ba901/auth-test-model-1ba901-kserve-6bc7cd9cd9-9d9pp" podUID="c5601047-51b4-4310-9289-580b8891746e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:35:31.854350 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:31.848673 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:35:31.854350 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:31.848805 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 30 13:35:31.854350 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:31.848851 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" podUID="b446da63-fda1-43e4-91b0-9a2d2a77b1a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 30 13:35:32.851632 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:32.851592 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 30 13:35:32.852758 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:32.852725 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" podUID="b446da63-fda1-43e4-91b0-9a2d2a77b1a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 30 13:35:33.842631 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:33.842594 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 30 13:35:33.843040 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:33.843007 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" podUID="b446da63-fda1-43e4-91b0-9a2d2a77b1a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 30 13:35:34.846287 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:34.842653 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 30 13:35:34.846287 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:34.842716 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" podUID="b446da63-fda1-43e4-91b0-9a2d2a77b1a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 30 13:35:35.851292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:35.844663 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 30 13:35:35.851292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:35.844763 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" podUID="b446da63-fda1-43e4-91b0-9a2d2a77b1a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 30 13:35:36.918471 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:36.918433 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:35:36.919087 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:36.919061 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:35:36.919886 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:36.919860 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:35:37.354228 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:35:37.354193 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:35:41.458595 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:41.458530 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" podUID="b0d31d29-93c6-4992-b3b7-1bb491b382fb" containerName="main" containerID="cri-o://5b17843d200f30bc3806479fb4bc0545eae029717d7eec98edba750477787d09" gracePeriod=45 Oct 30 13:35:41.472437 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:41.472400 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" podUID="b0d31d29-93c6-4992-b3b7-1bb491b382fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": dial tcp 10.132.0.212:8000: connect: connection refused" Oct 30 13:35:41.760604 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:41.760581 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" Oct 30 13:35:41.851880 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:41.851833 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0d31d29-93c6-4992-b3b7-1bb491b382fb-home\" (UniqueName: \"kubernetes.io/empty-dir/b0d31d29-93c6-4992-b3b7-1bb491b382fb-home\") pod \"b0d31d29-93c6-4992-b3b7-1bb491b382fb\" (UID: \"b0d31d29-93c6-4992-b3b7-1bb491b382fb\") " Oct 30 13:35:41.852065 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:41.852048 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0d31d29-93c6-4992-b3b7-1bb491b382fb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d31d29-93c6-4992-b3b7-1bb491b382fb-tmp-dir\") pod \"b0d31d29-93c6-4992-b3b7-1bb491b382fb\" (UID: \"b0d31d29-93c6-4992-b3b7-1bb491b382fb\") " Oct 30 13:35:41.852124 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:41.852083 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0d31d29-93c6-4992-b3b7-1bb491b382fb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d31d29-93c6-4992-b3b7-1bb491b382fb-dshm\") pod \"b0d31d29-93c6-4992-b3b7-1bb491b382fb\" (UID: \"b0d31d29-93c6-4992-b3b7-1bb491b382fb\") " Oct 30 13:35:41.852186 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:41.852124 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d31d29-93c6-4992-b3b7-1bb491b382fb-home" pod "b0d31d29-93c6-4992-b3b7-1bb491b382fb" (UID: "b0d31d29-93c6-4992-b3b7-1bb491b382fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:41.852186 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:41.852155 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0d31d29-93c6-4992-b3b7-1bb491b382fb-tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d31d29-93c6-4992-b3b7-1bb491b382fb-tls-certs\") pod \"b0d31d29-93c6-4992-b3b7-1bb491b382fb\" (UID: \"b0d31d29-93c6-4992-b3b7-1bb491b382fb\") " Oct 30 13:35:41.852360 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:41.852336 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0d31d29-93c6-4992-b3b7-1bb491b382fb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d31d29-93c6-4992-b3b7-1bb491b382fb-model-cache\") pod \"b0d31d29-93c6-4992-b3b7-1bb491b382fb\" (UID: \"b0d31d29-93c6-4992-b3b7-1bb491b382fb\") " Oct 30 13:35:41.852432 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:41.852349 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d31d29-93c6-4992-b3b7-1bb491b382fb-tmp-dir" pod "b0d31d29-93c6-4992-b3b7-1bb491b382fb" (UID: "b0d31d29-93c6-4992-b3b7-1bb491b382fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:41.852432 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:41.852377 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0d31d29-93c6-4992-b3b7-1bb491b382fb-kube-api-access-csj5c\" (UniqueName: \"kubernetes.io/projected/b0d31d29-93c6-4992-b3b7-1bb491b382fb-kube-api-access-csj5c\") pod \"b0d31d29-93c6-4992-b3b7-1bb491b382fb\" (UID: \"b0d31d29-93c6-4992-b3b7-1bb491b382fb\") " Oct 30 13:35:41.853079 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:41.853051 4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d31d29-93c6-4992-b3b7-1bb491b382fb-home\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:35:41.853210 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:41.853081 4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d31d29-93c6-4992-b3b7-1bb491b382fb-tmp-dir\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:35:41.853683 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:41.853657 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d31d29-93c6-4992-b3b7-1bb491b382fb-model-cache" pod "b0d31d29-93c6-4992-b3b7-1bb491b382fb" (UID: "b0d31d29-93c6-4992-b3b7-1bb491b382fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:41.855001 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:41.854978 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0d31d29-93c6-4992-b3b7-1bb491b382fb-tls-certs" pod "b0d31d29-93c6-4992-b3b7-1bb491b382fb" (UID: "b0d31d29-93c6-4992-b3b7-1bb491b382fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:35:41.855110 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:41.855028 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d31d29-93c6-4992-b3b7-1bb491b382fb-dshm" pod "b0d31d29-93c6-4992-b3b7-1bb491b382fb" (UID: "b0d31d29-93c6-4992-b3b7-1bb491b382fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:41.858704 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:41.858684 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0d31d29-93c6-4992-b3b7-1bb491b382fb-kube-api-access-csj5c" pod "b0d31d29-93c6-4992-b3b7-1bb491b382fb" (UID: "b0d31d29-93c6-4992-b3b7-1bb491b382fb"). InnerVolumeSpecName "kube-api-access-csj5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:35:41.954617 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:41.954585 4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d31d29-93c6-4992-b3b7-1bb491b382fb-tls-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:35:41.954617 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:41.954611 4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d31d29-93c6-4992-b3b7-1bb491b382fb-model-cache\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:35:41.954617 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:41.954621 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csj5c\" (UniqueName: \"kubernetes.io/projected/b0d31d29-93c6-4992-b3b7-1bb491b382fb-kube-api-access-csj5c\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:35:41.954827 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:41.954631 4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d31d29-93c6-4992-b3b7-1bb491b382fb-dshm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:35:42.196411 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:42.196378 4931 generic.go:356] "Generic (PLEG): container finished" podID="b0d31d29-93c6-4992-b3b7-1bb491b382fb" containerID="5b17843d200f30bc3806479fb4bc0545eae029717d7eec98edba750477787d09" exitCode=0 Oct 30 13:35:42.196916 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:42.196892 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" event={"ID":"b0d31d29-93c6-4992-b3b7-1bb491b382fb","Type":"ContainerDied","Data":"5b17843d200f30bc3806479fb4bc0545eae029717d7eec98edba750477787d09"} Oct 30 13:35:42.197063 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:42.197047 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" event={"ID":"b0d31d29-93c6-4992-b3b7-1bb491b382fb","Type":"ContainerDied","Data":"83da0a8fb4e9d5e15641f6fa4beca7444e2e368d086f40469b895dad352b44d2"} Oct 30 13:35:42.197177 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:42.197165 4931 scope.go:122] "RemoveContainer" containerID="5b17843d200f30bc3806479fb4bc0545eae029717d7eec98edba750477787d09" Oct 30 13:35:42.197982 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:42.197956 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts" Oct 30 13:35:42.231542 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:42.231369 4931 scope.go:122] "RemoveContainer" containerID="5b17843d200f30bc3806479fb4bc0545eae029717d7eec98edba750477787d09" Oct 30 13:35:42.231892 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:35:42.231779 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b17843d200f30bc3806479fb4bc0545eae029717d7eec98edba750477787d09\": container with ID starting with 5b17843d200f30bc3806479fb4bc0545eae029717d7eec98edba750477787d09 not found: ID does not exist" containerID="5b17843d200f30bc3806479fb4bc0545eae029717d7eec98edba750477787d09" Oct 30 13:35:42.231892 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:42.231824 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b17843d200f30bc3806479fb4bc0545eae029717d7eec98edba750477787d09"} err="failed to get container status \"5b17843d200f30bc3806479fb4bc0545eae029717d7eec98edba750477787d09\": rpc error: code = NotFound desc = could not find container \"5b17843d200f30bc3806479fb4bc0545eae029717d7eec98edba750477787d09\": container with ID starting with 5b17843d200f30bc3806479fb4bc0545eae029717d7eec98edba750477787d09 not found: ID does not exist" Oct 30 13:35:42.264751 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:42.264720 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts"] Oct 30 13:35:42.270178 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:42.270147 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-627dfa/rate-test-model-627dfa-kserve-d56cb8b85-4f2ts"] Oct 30 13:35:43.515329 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:43.515292 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0d31d29-93c6-4992-b3b7-1bb491b382fb" path="/var/lib/kubelet/pods/b0d31d29-93c6-4992-b3b7-1bb491b382fb/volumes" Oct 30 13:35:52.032352 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:52.032309 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adf49f94-tqqq8"] Oct 30 13:35:52.047808 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:52.047774 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adf49f94-tqqq8"] Oct 30 13:35:53.514519 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:35:53.514443 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1464a40b-fa02-47d9-8f74-3061dedf2390" path="/var/lib/kubelet/pods/1464a40b-fa02-47d9-8f74-3061dedf2390/volumes" Oct 30 13:36:01.566595 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:01.566560 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:36:03.097437 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:03.097339 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3653b5e-kzxvr"] Oct 30 13:36:03.120003 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:03.119965 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3653b5e-kzxvr"] Oct 30 13:36:03.570045 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:03.570005 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9b64b32-aa7c-4448-bb79-3e93cdfc528a" path="/var/lib/kubelet/pods/b9b64b32-aa7c-4448-bb79-3e93cdfc528a/volumes" Oct 30 13:36:03.612493 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:03.612432 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" podUID="ee8505de-cac0-4e49-bd15-a044536b10e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:36:03.792718 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:03.792683 4931 scope.go:122] "RemoveContainer" containerID="5afffceba04b175616d91d7137f3617f84b1fc774f13329650b5573b7cdbee0d" Oct 30 13:36:03.817893 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:03.817848 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" podUID="606b252b-881c-4d40-8ba9-fc4ed937a995" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:36:03.834533 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:03.834495 4931 scope.go:122] "RemoveContainer" containerID="c9c418151585261061be2736ce244777dab22d4e60b5ebcb232f19e9397384b0" Oct 30 13:36:03.865087 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:03.865048 4931 scope.go:122] "RemoveContainer" containerID="a2101968fc63fa24074512a15d582be91d2fb5c020ff6c32058e82da8ad07edc" Oct 30 13:36:03.890472 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:03.888812 4931 scope.go:122] "RemoveContainer" containerID="0b95ea9f858bb956f5c08777648f57d9ae4fc09fd98325dd5f3d786573f01475" Oct 30 13:36:03.913918 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:03.913477 4931 scope.go:122] "RemoveContainer" containerID="5a72eeb3a0ae9faec2e0c1f0f4cd7d1418e3c46d3dac3e821514e5abdb04c99b" Oct 30 13:36:03.969192 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:03.968024 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" podUID="9577f571-30cd-4043-a3e4-72bf95837552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:36:07.353710 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:36:07.353677 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:36:30.808777 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:30.808736 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h"] Oct 30 13:36:30.809513 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:30.808955 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" podUID="721beeba-d88a-4b37-b2a0-191c954a008d" containerName="istio-proxy" containerID="cri-o://7609a223a613c24eb8b3909536a8c582f7194c689837e02a53050725620409fe" gracePeriod=30 Oct 30 13:36:30.820977 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:30.820946 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl"] Oct 30 13:36:30.821184 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:30.821161 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" podUID="0055d65c-ebe0-4c48-964f-36a4c7e8e59a" containerName="istio-proxy" containerID="cri-o://2442b59de64ba0280a4b9f3e192ccb71926dd14625e7c9d7720f455650f7003d" gracePeriod=30 Oct 30 13:36:33.621518 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:33.621476 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:36:33.622069 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:33.621708 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" podUID="0055d65c-ebe0-4c48-964f-36a4c7e8e59a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:36:33.622363 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:33.622342 4931 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:36:33.622454 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:33.622400 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" podUID="721beeba-d88a-4b37-b2a0-191c954a008d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:36:36.128197 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.127574 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" Oct 30 13:36:36.225980 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.225941 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj"] Oct 30 13:36:36.227360 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.227326 4931 cpu_manager.go:414] "containerMap: removing container" podUID="6972ffa9-c402-415f-8166-7db0e5d8097c" containerName="main" Oct 30 13:36:36.227360 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.227358 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6972ffa9-c402-415f-8166-7db0e5d8097c" containerName="main" Oct 30 13:36:36.227591 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.227402 4931 cpu_manager.go:414] "containerMap: removing container" podUID="b0d31d29-93c6-4992-b3b7-1bb491b382fb" containerName="main" Oct 30 13:36:36.227591 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.227410 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0d31d29-93c6-4992-b3b7-1bb491b382fb" containerName="main" Oct 30 13:36:36.227591 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.227433 4931 cpu_manager.go:414] "containerMap: removing container" podUID="0055d65c-ebe0-4c48-964f-36a4c7e8e59a" containerName="istio-proxy" Oct 30 13:36:36.227591 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.227443 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0055d65c-ebe0-4c48-964f-36a4c7e8e59a" containerName="istio-proxy" Oct 30 13:36:36.227591 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.227493 4931 cpu_manager.go:414] "containerMap: removing container" podUID="c5601047-51b4-4310-9289-580b8891746e" containerName="main" Oct 30 13:36:36.227591 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.227500 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5601047-51b4-4310-9289-580b8891746e" containerName="main" Oct 30 13:36:36.227911 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.227640 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5601047-51b4-4310-9289-580b8891746e" containerName="main" Oct 30 13:36:36.227911 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.227655 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6972ffa9-c402-415f-8166-7db0e5d8097c" containerName="main" Oct 30 13:36:36.227911 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.227669 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0d31d29-93c6-4992-b3b7-1bb491b382fb" containerName="main" Oct 30 13:36:36.227911 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.227681 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="0055d65c-ebe0-4c48-964f-36a4c7e8e59a" containerName="istio-proxy" Oct 30 13:36:36.230137 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.229443 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-istio-token\" (UniqueName: \"kubernetes.io/projected/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-istio-token\") pod \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " Oct 30 13:36:36.230137 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.229501 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-istio-data\") pod \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " Oct 30 13:36:36.230137 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.229605 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-istiod-ca-cert\") pod \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " Oct 30 13:36:36.230137 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.229674 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-workload-certs\") pod \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " Oct 30 13:36:36.230137 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.229774 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-istio-podinfo\") pod \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " Oct 30 13:36:36.230137 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.229916 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-credential-socket\") pod \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " Oct 30 13:36:36.230137 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.229955 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-istio-envoy\") pod \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " Oct 30 13:36:36.230137 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.230032 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-workload-socket\") pod \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " Oct 30 13:36:36.232076 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.230855 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-kube-api-access-dcg22\" (UniqueName: \"kubernetes.io/projected/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-kube-api-access-dcg22\") pod \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\" (UID: \"0055d65c-ebe0-4c48-964f-36a4c7e8e59a\") " Oct 30 13:36:36.233090 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.232909 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-istio-data" pod "0055d65c-ebe0-4c48-964f-36a4c7e8e59a" (UID: "0055d65c-ebe0-4c48-964f-36a4c7e8e59a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:36:36.233791 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.233426 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-credential-socket" pod "0055d65c-ebe0-4c48-964f-36a4c7e8e59a" (UID: "0055d65c-ebe0-4c48-964f-36a4c7e8e59a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:36:36.233791 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.233509 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-istiod-ca-cert" pod "0055d65c-ebe0-4c48-964f-36a4c7e8e59a" (UID: "0055d65c-ebe0-4c48-964f-36a4c7e8e59a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:36:36.234058 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.233870 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-workload-socket" pod "0055d65c-ebe0-4c48-964f-36a4c7e8e59a" (UID: "0055d65c-ebe0-4c48-964f-36a4c7e8e59a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:36:36.235628 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.235604 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-workload-certs" pod "0055d65c-ebe0-4c48-964f-36a4c7e8e59a" (UID: "0055d65c-ebe0-4c48-964f-36a4c7e8e59a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:36:36.239907 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.239868 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-istio-token" pod "0055d65c-ebe0-4c48-964f-36a4c7e8e59a" (UID: "0055d65c-ebe0-4c48-964f-36a4c7e8e59a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:36:36.264399 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.257519 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" Oct 30 13:36:36.265248 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.265216 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj"] Oct 30 13:36:36.293358 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.289603 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-3b47e7-openshift-default-dockercfg-fkvcj\"" Oct 30 13:36:36.293358 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.292460 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-istio-podinfo" pod "0055d65c-ebe0-4c48-964f-36a4c7e8e59a" (UID: "0055d65c-ebe0-4c48-964f-36a4c7e8e59a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:36:36.294802 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.294774 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-istio-envoy" pod "0055d65c-ebe0-4c48-964f-36a4c7e8e59a" (UID: "0055d65c-ebe0-4c48-964f-36a4c7e8e59a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:36:36.308981 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.308933 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-kube-api-access-dcg22" pod "0055d65c-ebe0-4c48-964f-36a4c7e8e59a" (UID: "0055d65c-ebe0-4c48-964f-36a4c7e8e59a"). InnerVolumeSpecName "kube-api-access-dcg22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:36:36.334440 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.333848 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:36:36.334440 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.333881 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:36:36.334440 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.333896 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:36:36.334440 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.333917 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:36:36.334440 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.333934 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcg22\" (UniqueName: \"kubernetes.io/projected/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-kube-api-access-dcg22\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:36:36.334440 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.333949 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:36:36.334440 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.333964 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:36:36.334440 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.333975 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:36:36.334440 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.333988 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0055d65c-ebe0-4c48-964f-36a4c7e8e59a-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:36:36.435479 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.435443 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnrnd\" (UniqueName: \"kubernetes.io/projected/c666cfb4-8b66-49b3-b248-2f2771088444-kube-api-access-gnrnd\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" Oct 30 13:36:36.435713 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.435503 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c666cfb4-8b66-49b3-b248-2f2771088444-workload-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" Oct 30 13:36:36.435713 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.435624 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c666cfb4-8b66-49b3-b248-2f2771088444-credential-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" Oct 30 13:36:36.435915 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.435765 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c666cfb4-8b66-49b3-b248-2f2771088444-istio-data\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" Oct 30 13:36:36.435915 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.435795 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c666cfb4-8b66-49b3-b248-2f2771088444-workload-certs\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" Oct 30 13:36:36.436043 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.435924 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c666cfb4-8b66-49b3-b248-2f2771088444-istio-podinfo\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" Oct 30 13:36:36.436043 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.435964 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c666cfb4-8b66-49b3-b248-2f2771088444-istio-envoy\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" Oct 30 13:36:36.436043 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.436039 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c666cfb4-8b66-49b3-b248-2f2771088444-istio-token\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" Oct 30 13:36:36.436216 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.436095 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c666cfb4-8b66-49b3-b248-2f2771088444-istiod-ca-cert\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" Oct 30 13:36:36.495238 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.495208 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" Oct 30 13:36:36.537369 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.537332 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnrnd\" (UniqueName: \"kubernetes.io/projected/c666cfb4-8b66-49b3-b248-2f2771088444-kube-api-access-gnrnd\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" Oct 30 13:36:36.537521 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.537384 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c666cfb4-8b66-49b3-b248-2f2771088444-workload-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" Oct 30 13:36:36.537521 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.537447 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c666cfb4-8b66-49b3-b248-2f2771088444-credential-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" Oct 30 13:36:36.537655 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.537557 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c666cfb4-8b66-49b3-b248-2f2771088444-istio-data\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" Oct 30 13:36:36.537655 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.537583 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c666cfb4-8b66-49b3-b248-2f2771088444-workload-certs\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" Oct 30 13:36:36.537655 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.537648 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c666cfb4-8b66-49b3-b248-2f2771088444-istio-podinfo\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" Oct 30 13:36:36.537836 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.537677 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c666cfb4-8b66-49b3-b248-2f2771088444-istio-envoy\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" Oct 30 13:36:36.537836 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.537706 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c666cfb4-8b66-49b3-b248-2f2771088444-istio-token\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" Oct 30 13:36:36.537836 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.537765 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c666cfb4-8b66-49b3-b248-2f2771088444-istiod-ca-cert\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" Oct 30 13:36:36.537836 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.537805 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c666cfb4-8b66-49b3-b248-2f2771088444-workload-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" Oct 30 13:36:36.538141 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.537870 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c666cfb4-8b66-49b3-b248-2f2771088444-credential-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" Oct 30 13:36:36.538141 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.537990 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c666cfb4-8b66-49b3-b248-2f2771088444-workload-certs\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" Oct 30 13:36:36.538247 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.538162 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c666cfb4-8b66-49b3-b248-2f2771088444-istio-data\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" Oct 30 13:36:36.538554 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.538527 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c666cfb4-8b66-49b3-b248-2f2771088444-istiod-ca-cert\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" Oct 30 13:36:36.540034 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.540000 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c666cfb4-8b66-49b3-b248-2f2771088444-istio-envoy\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" Oct 30 13:36:36.540140 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.540120 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c666cfb4-8b66-49b3-b248-2f2771088444-istio-podinfo\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" Oct 30 13:36:36.557175 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.557152 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd"] Oct 30 13:36:36.557846 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.557831 4931 cpu_manager.go:414] "containerMap: removing container" podUID="721beeba-d88a-4b37-b2a0-191c954a008d" containerName="istio-proxy" Oct 30 13:36:36.557901 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.557849 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="721beeba-d88a-4b37-b2a0-191c954a008d" containerName="istio-proxy" Oct 30 13:36:36.557994 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.557985 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="721beeba-d88a-4b37-b2a0-191c954a008d" containerName="istio-proxy" Oct 30 13:36:36.562448 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.562425 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnrnd\" (UniqueName: \"kubernetes.io/projected/c666cfb4-8b66-49b3-b248-2f2771088444-kube-api-access-gnrnd\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" Oct 30 13:36:36.562568 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.562545 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c666cfb4-8b66-49b3-b248-2f2771088444-istio-token\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" Oct 30 13:36:36.563078 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.563053 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" Oct 30 13:36:36.588356 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.588331 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd"] Oct 30 13:36:36.618236 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.618197 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" Oct 30 13:36:36.639080 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.639050 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/721beeba-d88a-4b37-b2a0-191c954a008d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/721beeba-d88a-4b37-b2a0-191c954a008d-istio-podinfo\") pod \"721beeba-d88a-4b37-b2a0-191c954a008d\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " Oct 30 13:36:36.639181 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.639156 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/721beeba-d88a-4b37-b2a0-191c954a008d-kube-api-access-v6w6m\" (UniqueName: \"kubernetes.io/projected/721beeba-d88a-4b37-b2a0-191c954a008d-kube-api-access-v6w6m\") pod \"721beeba-d88a-4b37-b2a0-191c954a008d\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " Oct 30 13:36:36.639251 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.639226 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/721beeba-d88a-4b37-b2a0-191c954a008d-istio-token\" (UniqueName: \"kubernetes.io/projected/721beeba-d88a-4b37-b2a0-191c954a008d-istio-token\") pod \"721beeba-d88a-4b37-b2a0-191c954a008d\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " Oct 30 13:36:36.639349 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.639283 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/721beeba-d88a-4b37-b2a0-191c954a008d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/721beeba-d88a-4b37-b2a0-191c954a008d-workload-certs\") pod \"721beeba-d88a-4b37-b2a0-191c954a008d\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " Oct 30 13:36:36.639406 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.639363 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/721beeba-d88a-4b37-b2a0-191c954a008d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/721beeba-d88a-4b37-b2a0-191c954a008d-istiod-ca-cert\") pod \"721beeba-d88a-4b37-b2a0-191c954a008d\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " Oct 30 13:36:36.639544 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.639529 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/721beeba-d88a-4b37-b2a0-191c954a008d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/721beeba-d88a-4b37-b2a0-191c954a008d-istio-envoy\") pod \"721beeba-d88a-4b37-b2a0-191c954a008d\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " Oct 30 13:36:36.639635 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.639622 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/721beeba-d88a-4b37-b2a0-191c954a008d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/721beeba-d88a-4b37-b2a0-191c954a008d-workload-socket\") pod \"721beeba-d88a-4b37-b2a0-191c954a008d\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " Oct 30 13:36:36.639694 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.639653 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/721beeba-d88a-4b37-b2a0-191c954a008d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/721beeba-d88a-4b37-b2a0-191c954a008d-istio-data\") pod \"721beeba-d88a-4b37-b2a0-191c954a008d\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " Oct 30 13:36:36.639755 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.639707 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/721beeba-d88a-4b37-b2a0-191c954a008d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/721beeba-d88a-4b37-b2a0-191c954a008d-credential-socket\") pod \"721beeba-d88a-4b37-b2a0-191c954a008d\" (UID: \"721beeba-d88a-4b37-b2a0-191c954a008d\") " Oct 30 13:36:36.639853 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.639823 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/721beeba-d88a-4b37-b2a0-191c954a008d-workload-certs" pod "721beeba-d88a-4b37-b2a0-191c954a008d" (UID: "721beeba-d88a-4b37-b2a0-191c954a008d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:36:36.640459 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.640433 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/721beeba-d88a-4b37-b2a0-191c954a008d-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:36:36.641623 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.641165 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/721beeba-d88a-4b37-b2a0-191c954a008d-istiod-ca-cert" pod "721beeba-d88a-4b37-b2a0-191c954a008d" (UID: "721beeba-d88a-4b37-b2a0-191c954a008d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:36:36.641883 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.641856 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/721beeba-d88a-4b37-b2a0-191c954a008d-istio-data" pod "721beeba-d88a-4b37-b2a0-191c954a008d" (UID: "721beeba-d88a-4b37-b2a0-191c954a008d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:36:36.642065 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.642041 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/721beeba-d88a-4b37-b2a0-191c954a008d-kube-api-access-v6w6m" pod "721beeba-d88a-4b37-b2a0-191c954a008d" (UID: "721beeba-d88a-4b37-b2a0-191c954a008d"). InnerVolumeSpecName "kube-api-access-v6w6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:36:36.642307 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.642243 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/721beeba-d88a-4b37-b2a0-191c954a008d-credential-socket" pod "721beeba-d88a-4b37-b2a0-191c954a008d" (UID: "721beeba-d88a-4b37-b2a0-191c954a008d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:36:36.642461 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.642441 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/721beeba-d88a-4b37-b2a0-191c954a008d-workload-socket" pod "721beeba-d88a-4b37-b2a0-191c954a008d" (UID: "721beeba-d88a-4b37-b2a0-191c954a008d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:36:36.642566 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.642455 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/721beeba-d88a-4b37-b2a0-191c954a008d-istio-podinfo" pod "721beeba-d88a-4b37-b2a0-191c954a008d" (UID: "721beeba-d88a-4b37-b2a0-191c954a008d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:36:36.644088 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.643816 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/721beeba-d88a-4b37-b2a0-191c954a008d-istio-token" pod "721beeba-d88a-4b37-b2a0-191c954a008d" (UID: "721beeba-d88a-4b37-b2a0-191c954a008d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:36:36.644088 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.643929 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/721beeba-d88a-4b37-b2a0-191c954a008d-istio-envoy" pod "721beeba-d88a-4b37-b2a0-191c954a008d" (UID: "721beeba-d88a-4b37-b2a0-191c954a008d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:36:36.741726 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.741695 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc46j\" (UniqueName: \"kubernetes.io/projected/6a38c538-ca9f-4135-96f8-32dff99e07d7-kube-api-access-dc46j\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" Oct 30 13:36:36.741926 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.741732 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a38c538-ca9f-4135-96f8-32dff99e07d7-istiod-ca-cert\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" Oct 30 13:36:36.742010 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.741938 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a38c538-ca9f-4135-96f8-32dff99e07d7-istio-token\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" Oct 30 13:36:36.742010 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.742002 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a38c538-ca9f-4135-96f8-32dff99e07d7-istio-podinfo\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" Oct 30 13:36:36.742123 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.742099 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a38c538-ca9f-4135-96f8-32dff99e07d7-credential-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" Oct 30 13:36:36.742189 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.742174 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a38c538-ca9f-4135-96f8-32dff99e07d7-istio-data\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" Oct 30 13:36:36.742390 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.742256 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a38c538-ca9f-4135-96f8-32dff99e07d7-istio-envoy\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" Oct 30 13:36:36.742390 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.742328 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a38c538-ca9f-4135-96f8-32dff99e07d7-workload-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" Oct 30 13:36:36.742390 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.742382 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a38c538-ca9f-4135-96f8-32dff99e07d7-workload-certs\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" Oct 30 13:36:36.742536 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.742487 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/721beeba-d88a-4b37-b2a0-191c954a008d-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:36:36.742536 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.742500 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6w6m\" (UniqueName: \"kubernetes.io/projected/721beeba-d88a-4b37-b2a0-191c954a008d-kube-api-access-v6w6m\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:36:36.742536 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.742511 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/721beeba-d88a-4b37-b2a0-191c954a008d-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:36:36.742536 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.742520 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/721beeba-d88a-4b37-b2a0-191c954a008d-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:36:36.742536 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.742530 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/721beeba-d88a-4b37-b2a0-191c954a008d-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:36:36.742701 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.742537 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/721beeba-d88a-4b37-b2a0-191c954a008d-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:36:36.742701 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.742546 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/721beeba-d88a-4b37-b2a0-191c954a008d-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:36:36.742701 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.742555 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/721beeba-d88a-4b37-b2a0-191c954a008d-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:36:36.822345 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.822308 4931 generic.go:356] "Generic (PLEG): container finished" podID="0055d65c-ebe0-4c48-964f-36a4c7e8e59a" containerID="2442b59de64ba0280a4b9f3e192ccb71926dd14625e7c9d7720f455650f7003d" exitCode=0 Oct 30 13:36:36.822527 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.822414 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" Oct 30 13:36:36.822527 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.822433 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" event={"ID":"0055d65c-ebe0-4c48-964f-36a4c7e8e59a","Type":"ContainerDied","Data":"2442b59de64ba0280a4b9f3e192ccb71926dd14625e7c9d7720f455650f7003d"} Oct 30 13:36:36.822651 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.822526 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl" event={"ID":"0055d65c-ebe0-4c48-964f-36a4c7e8e59a","Type":"ContainerDied","Data":"efacfc5f681721aec5f25ae008b914a88075b57bd576da5cdc2888dce6c15772"} Oct 30 13:36:36.822651 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.822556 4931 scope.go:122] "RemoveContainer" containerID="2442b59de64ba0280a4b9f3e192ccb71926dd14625e7c9d7720f455650f7003d" Oct 30 13:36:36.824178 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.824153 4931 generic.go:356] "Generic (PLEG): container finished" podID="721beeba-d88a-4b37-b2a0-191c954a008d" containerID="7609a223a613c24eb8b3909536a8c582f7194c689837e02a53050725620409fe" exitCode=0 Oct 30 13:36:36.824366 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.824306 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" event={"ID":"721beeba-d88a-4b37-b2a0-191c954a008d","Type":"ContainerDied","Data":"7609a223a613c24eb8b3909536a8c582f7194c689837e02a53050725620409fe"} Oct 30 13:36:36.824366 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.824352 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" Oct 30 13:36:36.824497 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.824382 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h" event={"ID":"721beeba-d88a-4b37-b2a0-191c954a008d","Type":"ContainerDied","Data":"63b34a81dcd50c04ddbfd90617c8d21609985ed01e90d40f28c8f46167667e6a"} Oct 30 13:36:36.843370 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.843340 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a38c538-ca9f-4135-96f8-32dff99e07d7-workload-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" Oct 30 13:36:36.843496 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.843410 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a38c538-ca9f-4135-96f8-32dff99e07d7-workload-certs\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" Oct 30 13:36:36.843496 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.843478 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc46j\" (UniqueName: \"kubernetes.io/projected/6a38c538-ca9f-4135-96f8-32dff99e07d7-kube-api-access-dc46j\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" Oct 30 13:36:36.843639 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.843508 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a38c538-ca9f-4135-96f8-32dff99e07d7-istiod-ca-cert\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" Oct 30 13:36:36.843639 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.843625 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a38c538-ca9f-4135-96f8-32dff99e07d7-istio-token\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" Oct 30 13:36:36.843773 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.843670 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a38c538-ca9f-4135-96f8-32dff99e07d7-istio-podinfo\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" Oct 30 13:36:36.843773 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.843711 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a38c538-ca9f-4135-96f8-32dff99e07d7-workload-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" Oct 30 13:36:36.843773 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.843738 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a38c538-ca9f-4135-96f8-32dff99e07d7-credential-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" Oct 30 13:36:36.843926 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.843764 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a38c538-ca9f-4135-96f8-32dff99e07d7-workload-certs\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" Oct 30 13:36:36.843926 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.843800 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a38c538-ca9f-4135-96f8-32dff99e07d7-istio-data\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" Oct 30 13:36:36.843926 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.843892 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a38c538-ca9f-4135-96f8-32dff99e07d7-istio-envoy\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" Oct 30 13:36:36.844178 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.844154 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a38c538-ca9f-4135-96f8-32dff99e07d7-istio-data\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" Oct 30 13:36:36.844459 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.844438 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a38c538-ca9f-4135-96f8-32dff99e07d7-istiod-ca-cert\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" Oct 30 13:36:36.844580 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.844536 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a38c538-ca9f-4135-96f8-32dff99e07d7-credential-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" Oct 30 13:36:36.846111 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.846091 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a38c538-ca9f-4135-96f8-32dff99e07d7-istio-envoy\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" Oct 30 13:36:36.846235 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.846216 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a38c538-ca9f-4135-96f8-32dff99e07d7-istio-podinfo\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" Oct 30 13:36:36.869103 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.869078 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc46j\" (UniqueName: \"kubernetes.io/projected/6a38c538-ca9f-4135-96f8-32dff99e07d7-kube-api-access-dc46j\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" Oct 30 13:36:36.869385 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.869364 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a38c538-ca9f-4135-96f8-32dff99e07d7-istio-token\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" Oct 30 13:36:36.869976 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.869958 4931 scope.go:122] "RemoveContainer" containerID="2442b59de64ba0280a4b9f3e192ccb71926dd14625e7c9d7720f455650f7003d" Oct 30 13:36:36.870317 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:36:36.870299 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2442b59de64ba0280a4b9f3e192ccb71926dd14625e7c9d7720f455650f7003d\": container with ID starting with 2442b59de64ba0280a4b9f3e192ccb71926dd14625e7c9d7720f455650f7003d not found: ID does not exist" containerID="2442b59de64ba0280a4b9f3e192ccb71926dd14625e7c9d7720f455650f7003d" Oct 30 13:36:36.870381 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.870327 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2442b59de64ba0280a4b9f3e192ccb71926dd14625e7c9d7720f455650f7003d"} err="failed to get container status \"2442b59de64ba0280a4b9f3e192ccb71926dd14625e7c9d7720f455650f7003d\": rpc error: code = NotFound desc = could not find container \"2442b59de64ba0280a4b9f3e192ccb71926dd14625e7c9d7720f455650f7003d\": container with ID starting with 2442b59de64ba0280a4b9f3e192ccb71926dd14625e7c9d7720f455650f7003d not found: ID does not exist" Oct 30 13:36:36.870381 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.870344 4931 scope.go:122] "RemoveContainer" containerID="7609a223a613c24eb8b3909536a8c582f7194c689837e02a53050725620409fe" Oct 30 13:36:36.878662 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.878642 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" Oct 30 13:36:36.914577 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.914538 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h"] Oct 30 13:36:36.920168 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.919236 4931 scope.go:122] "RemoveContainer" containerID="7609a223a613c24eb8b3909536a8c582f7194c689837e02a53050725620409fe" Oct 30 13:36:36.920168 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.919674 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h"] Oct 30 13:36:36.920168 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:36:36.919786 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7609a223a613c24eb8b3909536a8c582f7194c689837e02a53050725620409fe\": container with ID starting with 7609a223a613c24eb8b3909536a8c582f7194c689837e02a53050725620409fe not found: ID does not exist" containerID="7609a223a613c24eb8b3909536a8c582f7194c689837e02a53050725620409fe" Oct 30 13:36:36.920168 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.919823 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7609a223a613c24eb8b3909536a8c582f7194c689837e02a53050725620409fe"} err="failed to get container status \"7609a223a613c24eb8b3909536a8c582f7194c689837e02a53050725620409fe\": rpc error: code = NotFound desc = could not find container \"7609a223a613c24eb8b3909536a8c582f7194c689837e02a53050725620409fe\": container with ID starting with 7609a223a613c24eb8b3909536a8c582f7194c689837e02a53050725620409fe not found: ID does not exist" Oct 30 13:36:36.963884 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.963846 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj"] Oct 30 13:36:36.966002 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:36:36.965966 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc666cfb4_8b66_49b3_b248_2f2771088444.slice/crio-dcc4dec218149db23ae3e0bf6b58e453c15bc65dbe35bd29459bb40a566e5db6 WatchSource:0}: Error finding container dcc4dec218149db23ae3e0bf6b58e453c15bc65dbe35bd29459bb40a566e5db6: Status 404 returned error can't find the container with id dcc4dec218149db23ae3e0bf6b58e453c15bc65dbe35bd29459bb40a566e5db6 Oct 30 13:36:36.966949 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.966809 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl"] Oct 30 13:36:36.970712 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.970666 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:36:36.970832 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.970753 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:36:36.970832 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.970801 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:36:36.971866 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:36.971571 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl"] Oct 30 13:36:37.211069 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:37.211036 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd"] Oct 30 13:36:37.212315 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:36:37.212285 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a38c538_ca9f_4135_96f8_32dff99e07d7.slice/crio-f0e2a66d0199fe2985a7edf099f6b8aa8564ab278ca4c7890331fee6ca688c6e WatchSource:0}: Error finding container f0e2a66d0199fe2985a7edf099f6b8aa8564ab278ca4c7890331fee6ca688c6e: Status 404 returned error can't find the container with id f0e2a66d0199fe2985a7edf099f6b8aa8564ab278ca4c7890331fee6ca688c6e Oct 30 13:36:37.215878 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:37.215838 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:36:37.215965 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:37.215934 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:36:37.216005 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:37.215981 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:36:37.367333 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:36:37.360317 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:36:37.516109 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:37.516026 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0055d65c-ebe0-4c48-964f-36a4c7e8e59a" path="/var/lib/kubelet/pods/0055d65c-ebe0-4c48-964f-36a4c7e8e59a/volumes" Oct 30 13:36:37.516642 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:37.516624 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="721beeba-d88a-4b37-b2a0-191c954a008d" path="/var/lib/kubelet/pods/721beeba-d88a-4b37-b2a0-191c954a008d/volumes" Oct 30 13:36:37.840329 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:37.840213 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" event={"ID":"c666cfb4-8b66-49b3-b248-2f2771088444","Type":"ContainerStarted","Data":"b1fbfd0a3711a27de8b03930069de51ec828290cf14a6961383d360484d02453"} Oct 30 13:36:37.840593 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:37.840567 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" event={"ID":"c666cfb4-8b66-49b3-b248-2f2771088444","Type":"ContainerStarted","Data":"dcc4dec218149db23ae3e0bf6b58e453c15bc65dbe35bd29459bb40a566e5db6"} Oct 30 13:36:37.844523 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:37.844473 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" event={"ID":"6a38c538-ca9f-4135-96f8-32dff99e07d7","Type":"ContainerStarted","Data":"094e0849ae68f5e03c375b4456a2242828b445abd1bc0e6d277926a5dcce6e3c"} Oct 30 13:36:37.844815 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:37.844791 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" event={"ID":"6a38c538-ca9f-4135-96f8-32dff99e07d7","Type":"ContainerStarted","Data":"f0e2a66d0199fe2985a7edf099f6b8aa8564ab278ca4c7890331fee6ca688c6e"} Oct 30 13:36:37.879070 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:37.879035 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" Oct 30 13:36:37.895456 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:37.895407 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" podStartSLOduration=15.895394234 podStartE2EDuration="15.895394234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:37.891941227 +0000 UTC m=+3096.495439705" watchObservedRunningTime="2026-10-30 13:36:37.895394234 +0000 UTC m=+3096.498892680" Oct 30 13:36:37.969968 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:37.969904 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" podStartSLOduration=60.969888922 podStartE2EDuration="1m0.969888922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:37.947825353 +0000 UTC m=+3096.551323801" watchObservedRunningTime="2026-10-30 13:36:37.969888922 +0000 UTC m=+3096.573387430" Oct 30 13:36:38.621311 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:38.621152 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" Oct 30 13:36:38.622436 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:38.622416 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 30 13:36:38.622508 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:38.622481 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" podUID="c666cfb4-8b66-49b3-b248-2f2771088444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 30 13:36:38.883389 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:38.882591 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 30 13:36:38.883389 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:38.882656 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" podUID="6a38c538-ca9f-4135-96f8-32dff99e07d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 30 13:36:39.620768 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:39.620731 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 30 13:36:39.620958 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:39.620785 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" podUID="c666cfb4-8b66-49b3-b248-2f2771088444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 30 13:36:39.883356 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:39.882748 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 30 13:36:39.883356 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:39.882826 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" podUID="6a38c538-ca9f-4135-96f8-32dff99e07d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 30 13:36:40.620672 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:40.620636 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 30 13:36:40.620858 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:40.620686 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" podUID="c666cfb4-8b66-49b3-b248-2f2771088444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 30 13:36:40.880001 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:40.879904 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 30 13:36:40.880427 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:40.880388 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" podUID="6a38c538-ca9f-4135-96f8-32dff99e07d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 30 13:36:41.620804 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:41.620772 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 30 13:36:41.621242 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:41.620822 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" podUID="c666cfb4-8b66-49b3-b248-2f2771088444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 30 13:36:41.879779 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:41.879684 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 30 13:36:41.880072 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:41.880036 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" podUID="6a38c538-ca9f-4135-96f8-32dff99e07d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 30 13:36:42.621199 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:42.621165 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 30 13:36:42.621674 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:42.621217 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" podUID="c666cfb4-8b66-49b3-b248-2f2771088444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 30 13:36:42.906314 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:42.895434 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 30 13:36:42.906662 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:42.906623 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" podUID="6a38c538-ca9f-4135-96f8-32dff99e07d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 30 13:36:43.620970 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:43.620919 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 30 13:36:43.621165 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:43.620977 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" podUID="c666cfb4-8b66-49b3-b248-2f2771088444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 30 13:36:43.879525 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:43.879443 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 30 13:36:43.879525 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:43.879501 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" podUID="6a38c538-ca9f-4135-96f8-32dff99e07d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 30 13:36:44.626253 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:44.626212 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" Oct 30 13:36:44.626726 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:44.626700 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" Oct 30 13:36:44.627545 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:44.627508 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" Oct 30 13:36:44.702427 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:44.702393 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:36:44.885709 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:44.885630 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" Oct 30 13:36:44.886153 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:44.886133 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" Oct 30 13:36:44.886853 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:36:44.886833 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" Oct 30 13:37:04.172724 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:04.172689 4931 scope.go:122] "RemoveContainer" containerID="e0729d33f298ee1898d98c0a37160429212f1a32c7580d0abeefa582187a0fd5" Oct 30 13:37:04.183331 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:04.183310 4931 scope.go:122] "RemoveContainer" containerID="aa45c69f3d282adbd98668437806ccc2ca8f0ab73436decf3cc427e2161a3332" Oct 30 13:37:07.359633 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:37:07.359586 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:37:22.534755 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:22.534720 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv"] Oct 30 13:37:22.535376 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:22.534915 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" podUID="7a33526d-ab5b-4e65-8111-b5a398514d40" containerName="istio-proxy" containerID="cri-o://523ec54bf6530048c5b57739eb0d3f8a12bc5af74b132f6122f9796a282b195e" gracePeriod=30 Oct 30 13:37:27.829739 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:27.829710 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" Oct 30 13:37:27.916835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:27.916799 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws"] Oct 30 13:37:27.917600 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:27.917576 4931 cpu_manager.go:414] "containerMap: removing container" podUID="7a33526d-ab5b-4e65-8111-b5a398514d40" containerName="istio-proxy" Oct 30 13:37:27.917600 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:27.917597 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a33526d-ab5b-4e65-8111-b5a398514d40" containerName="istio-proxy" Oct 30 13:37:27.917768 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:27.917723 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a33526d-ab5b-4e65-8111-b5a398514d40" containerName="istio-proxy" Oct 30 13:37:27.918832 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:27.918805 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a33526d-ab5b-4e65-8111-b5a398514d40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a33526d-ab5b-4e65-8111-b5a398514d40-credential-socket\") pod \"7a33526d-ab5b-4e65-8111-b5a398514d40\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " Oct 30 13:37:27.918993 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:27.918979 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a33526d-ab5b-4e65-8111-b5a398514d40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a33526d-ab5b-4e65-8111-b5a398514d40-istio-envoy\") pod \"7a33526d-ab5b-4e65-8111-b5a398514d40\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " Oct 30 13:37:27.919051 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:27.919027 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a33526d-ab5b-4e65-8111-b5a398514d40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a33526d-ab5b-4e65-8111-b5a398514d40-istiod-ca-cert\") pod \"7a33526d-ab5b-4e65-8111-b5a398514d40\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " Oct 30 13:37:27.919110 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:27.919074 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a33526d-ab5b-4e65-8111-b5a398514d40-kube-api-access-fdk77\" (UniqueName: \"kubernetes.io/projected/7a33526d-ab5b-4e65-8111-b5a398514d40-kube-api-access-fdk77\") pod \"7a33526d-ab5b-4e65-8111-b5a398514d40\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " Oct 30 13:37:27.919174 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:27.919144 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a33526d-ab5b-4e65-8111-b5a398514d40-credential-socket" pod "7a33526d-ab5b-4e65-8111-b5a398514d40" (UID: "7a33526d-ab5b-4e65-8111-b5a398514d40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:27.919546 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:27.919525 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a33526d-ab5b-4e65-8111-b5a398514d40-istiod-ca-cert" pod "7a33526d-ab5b-4e65-8111-b5a398514d40" (UID: "7a33526d-ab5b-4e65-8111-b5a398514d40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:37:27.919668 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:27.919630 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a33526d-ab5b-4e65-8111-b5a398514d40-istio-token\" (UniqueName: \"kubernetes.io/projected/7a33526d-ab5b-4e65-8111-b5a398514d40-istio-token\") pod \"7a33526d-ab5b-4e65-8111-b5a398514d40\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " Oct 30 13:37:27.919733 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:27.919690 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a33526d-ab5b-4e65-8111-b5a398514d40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a33526d-ab5b-4e65-8111-b5a398514d40-workload-certs\") pod \"7a33526d-ab5b-4e65-8111-b5a398514d40\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " Oct 30 13:37:27.920816 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:27.919817 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a33526d-ab5b-4e65-8111-b5a398514d40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a33526d-ab5b-4e65-8111-b5a398514d40-istio-podinfo\") pod \"7a33526d-ab5b-4e65-8111-b5a398514d40\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " Oct 30 13:37:27.920816 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:27.919855 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a33526d-ab5b-4e65-8111-b5a398514d40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a33526d-ab5b-4e65-8111-b5a398514d40-istio-data\") pod \"7a33526d-ab5b-4e65-8111-b5a398514d40\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " Oct 30 13:37:27.920816 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:27.920085 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a33526d-ab5b-4e65-8111-b5a398514d40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a33526d-ab5b-4e65-8111-b5a398514d40-workload-socket\") pod \"7a33526d-ab5b-4e65-8111-b5a398514d40\" (UID: \"7a33526d-ab5b-4e65-8111-b5a398514d40\") " Oct 30 13:37:27.920816 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:27.920494 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a33526d-ab5b-4e65-8111-b5a398514d40-istio-data" pod "7a33526d-ab5b-4e65-8111-b5a398514d40" (UID: "7a33526d-ab5b-4e65-8111-b5a398514d40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:27.920816 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:27.920568 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a33526d-ab5b-4e65-8111-b5a398514d40-workload-certs" pod "7a33526d-ab5b-4e65-8111-b5a398514d40" (UID: "7a33526d-ab5b-4e65-8111-b5a398514d40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:27.921159 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:27.920823 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a33526d-ab5b-4e65-8111-b5a398514d40-workload-socket" pod "7a33526d-ab5b-4e65-8111-b5a398514d40" (UID: "7a33526d-ab5b-4e65-8111-b5a398514d40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:27.921675 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:27.921645 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a33526d-ab5b-4e65-8111-b5a398514d40-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:27.921784 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:27.921680 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a33526d-ab5b-4e65-8111-b5a398514d40-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:27.921784 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:27.921693 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a33526d-ab5b-4e65-8111-b5a398514d40-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:27.921784 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:27.921705 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a33526d-ab5b-4e65-8111-b5a398514d40-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:27.921784 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:27.921718 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a33526d-ab5b-4e65-8111-b5a398514d40-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:27.922123 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:27.921963 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" Oct 30 13:37:27.923599 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:27.923574 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a33526d-ab5b-4e65-8111-b5a398514d40-kube-api-access-fdk77" pod "7a33526d-ab5b-4e65-8111-b5a398514d40" (UID: "7a33526d-ab5b-4e65-8111-b5a398514d40"). InnerVolumeSpecName "kube-api-access-fdk77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:37:27.923738 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:27.923606 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a33526d-ab5b-4e65-8111-b5a398514d40-istio-token" pod "7a33526d-ab5b-4e65-8111-b5a398514d40" (UID: "7a33526d-ab5b-4e65-8111-b5a398514d40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:37:27.923738 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:27.923655 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a33526d-ab5b-4e65-8111-b5a398514d40-istio-envoy" pod "7a33526d-ab5b-4e65-8111-b5a398514d40" (UID: "7a33526d-ab5b-4e65-8111-b5a398514d40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:27.923738 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:27.923719 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a33526d-ab5b-4e65-8111-b5a398514d40-istio-podinfo" pod "7a33526d-ab5b-4e65-8111-b5a398514d40" (UID: "7a33526d-ab5b-4e65-8111-b5a398514d40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:37:27.942386 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:27.942362 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-627dfa-openshift-default-dockercfg-mnpmk\"" Oct 30 13:37:27.960212 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:27.960182 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws"] Oct 30 13:37:28.053307 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.053237 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a33526d-ab5b-4e65-8111-b5a398514d40-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:28.053650 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.053624 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdk77\" (UniqueName: \"kubernetes.io/projected/7a33526d-ab5b-4e65-8111-b5a398514d40-kube-api-access-fdk77\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:28.053770 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.053652 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a33526d-ab5b-4e65-8111-b5a398514d40-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:28.053770 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.053668 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a33526d-ab5b-4e65-8111-b5a398514d40-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:28.159339 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.155387 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-istiod-ca-cert\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" Oct 30 13:37:28.159339 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.155526 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-workload-certs\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" Oct 30 13:37:28.159339 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.155629 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-istio-envoy\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" Oct 30 13:37:28.159339 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.155707 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-credential-socket\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" Oct 30 13:37:28.159339 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.155775 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qbvq\" (UniqueName: \"kubernetes.io/projected/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-kube-api-access-8qbvq\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" Oct 30 13:37:28.159339 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.155845 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-istio-token\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" Oct 30 13:37:28.159339 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.155979 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-istio-podinfo\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" Oct 30 13:37:28.159339 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.156040 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-istio-data\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" Oct 30 13:37:28.159339 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.156090 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-workload-socket\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" Oct 30 13:37:28.257487 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.257450 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-workload-certs\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" Oct 30 13:37:28.257694 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.257613 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-istio-envoy\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" Oct 30 13:37:28.257694 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.257654 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-credential-socket\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" Oct 30 13:37:28.257836 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.257693 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qbvq\" (UniqueName: \"kubernetes.io/projected/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-kube-api-access-8qbvq\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" Oct 30 13:37:28.257836 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.257746 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-istio-token\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" Oct 30 13:37:28.257963 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.257845 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-istio-podinfo\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" Oct 30 13:37:28.257963 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.257886 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-istio-data\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" Oct 30 13:37:28.257963 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.257938 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-workload-socket\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" Oct 30 13:37:28.257963 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.257950 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-workload-certs\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" Oct 30 13:37:28.258187 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.258105 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-istiod-ca-cert\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" Oct 30 13:37:28.258187 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.258133 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-credential-socket\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" Oct 30 13:37:28.258331 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.258232 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-istio-data\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" Oct 30 13:37:28.258384 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.258334 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-workload-socket\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" Oct 30 13:37:28.258753 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.258732 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-istiod-ca-cert\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" Oct 30 13:37:28.260063 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.260042 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-istio-envoy\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" Oct 30 13:37:28.260475 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.260445 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-istio-podinfo\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" Oct 30 13:37:28.283940 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.283743 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qbvq\" (UniqueName: \"kubernetes.io/projected/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-kube-api-access-8qbvq\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" Oct 30 13:37:28.284847 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.284822 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-istio-token\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" Oct 30 13:37:28.576353 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.576316 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" Oct 30 13:37:28.577415 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.577385 4931 generic.go:356] "Generic (PLEG): container finished" podID="7a33526d-ab5b-4e65-8111-b5a398514d40" containerID="523ec54bf6530048c5b57739eb0d3f8a12bc5af74b132f6122f9796a282b195e" exitCode=0 Oct 30 13:37:28.577546 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.577438 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" Oct 30 13:37:28.577546 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.577476 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" event={"ID":"7a33526d-ab5b-4e65-8111-b5a398514d40","Type":"ContainerDied","Data":"523ec54bf6530048c5b57739eb0d3f8a12bc5af74b132f6122f9796a282b195e"} Oct 30 13:37:28.577546 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.577534 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv" event={"ID":"7a33526d-ab5b-4e65-8111-b5a398514d40","Type":"ContainerDied","Data":"1c32cf5cc2ce7d2a2b7917713fa9dd3e387feb1ce3edf3f31fc6afb659a458c8"} Oct 30 13:37:28.577656 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.577560 4931 scope.go:122] "RemoveContainer" containerID="523ec54bf6530048c5b57739eb0d3f8a12bc5af74b132f6122f9796a282b195e" Oct 30 13:37:28.592460 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.592438 4931 scope.go:122] "RemoveContainer" containerID="523ec54bf6530048c5b57739eb0d3f8a12bc5af74b132f6122f9796a282b195e" Oct 30 13:37:28.592749 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:37:28.592724 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"523ec54bf6530048c5b57739eb0d3f8a12bc5af74b132f6122f9796a282b195e\": container with ID starting with 523ec54bf6530048c5b57739eb0d3f8a12bc5af74b132f6122f9796a282b195e not found: ID does not exist" containerID="523ec54bf6530048c5b57739eb0d3f8a12bc5af74b132f6122f9796a282b195e" Oct 30 13:37:28.592819 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.592757 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"523ec54bf6530048c5b57739eb0d3f8a12bc5af74b132f6122f9796a282b195e"} err="failed to get container status \"523ec54bf6530048c5b57739eb0d3f8a12bc5af74b132f6122f9796a282b195e\": rpc error: code = NotFound desc = could not find container \"523ec54bf6530048c5b57739eb0d3f8a12bc5af74b132f6122f9796a282b195e\": container with ID starting with 523ec54bf6530048c5b57739eb0d3f8a12bc5af74b132f6122f9796a282b195e not found: ID does not exist" Oct 30 13:37:28.642203 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.642173 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv"] Oct 30 13:37:28.655722 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.655690 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv"] Oct 30 13:37:28.706420 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.706365 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7"] Oct 30 13:37:28.706621 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.706598 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" podUID="a9babe7d-51a7-4d00-ab49-76441dd6dec5" containerName="istio-proxy" containerID="cri-o://2e3198b0df152359f3a8fb485a11d9f4c9f2d5ecb996eaae31ee1d0b009db496" gracePeriod=30 Oct 30 13:37:28.714549 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.714519 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht"] Oct 30 13:37:28.714732 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.714711 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" podUID="cf83a441-04d6-46bc-a81d-1c47f8a68e8e" containerName="istio-proxy" containerID="cri-o://830343a1f2bbfbd0ee3a784909a82cacc2fd9005ffa15706b14d001645fb5816" gracePeriod=30 Oct 30 13:37:28.723425 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.722792 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq"] Oct 30 13:37:28.723425 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:28.723003 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" podUID="6d987afd-6dc8-401b-a339-3909dff59665" containerName="istio-proxy" containerID="cri-o://4bdcfb97e3fe87e7eea3b0d9d44cea2ede26f7e1db3835cf001a958f272cf289" gracePeriod=30 Oct 30 13:37:29.099812 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:29.099778 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws"] Oct 30 13:37:29.103304 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:29.103246 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:37:29.103464 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:29.103355 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:37:29.103464 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:29.103404 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:37:29.476169 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:29.476132 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr"] Oct 30 13:37:29.476906 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:29.476544 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr" podUID="2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc" containerName="maas-api" containerID="cri-o://3a37f056b790f90747f3a268edcd83510ca4ade4b5f6dcb8c31ed52a378c8a8b" gracePeriod=30 Oct 30 13:37:29.538605 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:29.538567 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a33526d-ab5b-4e65-8111-b5a398514d40" path="/var/lib/kubelet/pods/7a33526d-ab5b-4e65-8111-b5a398514d40/volumes" Oct 30 13:37:29.595356 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:29.591348 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" event={"ID":"e46deba0-1ef3-4b54-9821-d4fbf394c9a6","Type":"ContainerStarted","Data":"bdd6ac7ab3e0c1e2e426a3c54fe195f84b2675d04e68c551dd1ea7314ca3c911"} Oct 30 13:37:29.595356 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:29.591434 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" event={"ID":"e46deba0-1ef3-4b54-9821-d4fbf394c9a6","Type":"ContainerStarted","Data":"4dbfb122067215b62beecc0e17c763d6c4fe586ad87830e936a1e21a7421127b"} Oct 30 13:37:29.654774 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:29.654699 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" podStartSLOduration=173.654682938 podStartE2EDuration="2m53.654682938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:29.652385733 +0000 UTC m=+3148.255884174" watchObservedRunningTime="2026-10-30 13:37:29.654682938 +0000 UTC m=+3148.258181386" Oct 30 13:37:30.220293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.206715 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr" Oct 30 13:37:30.323343 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.323179 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc-metrics-certs\") pod \"2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc\" (UID: \"2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc\") " Oct 30 13:37:30.323343 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.323239 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc-kube-api-access-x7wsk\" (UniqueName: \"kubernetes.io/projected/2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc-kube-api-access-x7wsk\") pod \"2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc\" (UID: \"2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc\") " Oct 30 13:37:30.323596 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.323372 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc-openshift-service-ca\") pod \"2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc\" (UID: \"2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc\") " Oct 30 13:37:30.323596 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.323525 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc-maas-api-tls\") pod \"2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc\" (UID: \"2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc\") " Oct 30 13:37:30.392863 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.392812 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc-maas-api-tls" pod "2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc" (UID: "2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:37:30.399488 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.395676 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc-openshift-service-ca" pod "2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc" (UID: "2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:37:30.412344 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.410295 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc-metrics-certs" pod "2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc" (UID: "2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:37:30.412344 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.410350 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc-kube-api-access-x7wsk" pod "2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc" (UID: "2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc"). InnerVolumeSpecName "kube-api-access-x7wsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:37:30.481132 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.481092 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7wsk\" (UniqueName: \"kubernetes.io/projected/2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc-kube-api-access-x7wsk\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:30.481390 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.481346 4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc-openshift-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:30.481390 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.481371 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc-maas-api-tls\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:30.481390 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.481383 4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc-metrics-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:30.491281 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.490850 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs"] Oct 30 13:37:30.495317 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.492053 4931 cpu_manager.go:414] "containerMap: removing container" podUID="2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc" containerName="maas-api" Oct 30 13:37:30.495317 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.492082 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc" containerName="maas-api" Oct 30 13:37:30.495317 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.492318 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc" containerName="maas-api" Oct 30 13:37:30.496911 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.496888 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs"] Oct 30 13:37:30.496998 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.496980 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs" Oct 30 13:37:30.518144 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.518109 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-72208f-serving-cert\"" Oct 30 13:37:30.576812 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.576735 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" Oct 30 13:37:30.578641 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.578591 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 13:37:30.578869 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.578647 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" podUID="e46deba0-1ef3-4b54-9821-d4fbf394c9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 13:37:30.582614 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.582584 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f409e7f-6b24-410e-896c-a8a557298edf-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs\" (UID: \"7f409e7f-6b24-410e-896c-a8a557298edf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs" Oct 30 13:37:30.582776 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.582757 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f409e7f-6b24-410e-896c-a8a557298edf-maas-api-tls\") pod \"maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs\" (UID: \"7f409e7f-6b24-410e-896c-a8a557298edf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs" Oct 30 13:37:30.582860 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.582842 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f409e7f-6b24-410e-896c-a8a557298edf-metrics-certs\") pod \"maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs\" (UID: \"7f409e7f-6b24-410e-896c-a8a557298edf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs" Oct 30 13:37:30.582915 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.582885 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6x6s\" (UniqueName: \"kubernetes.io/projected/7f409e7f-6b24-410e-896c-a8a557298edf-kube-api-access-d6x6s\") pod \"maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs\" (UID: \"7f409e7f-6b24-410e-896c-a8a557298edf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs" Oct 30 13:37:30.600976 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.600947 4931 generic.go:356] "Generic (PLEG): container finished" podID="2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc" containerID="3a37f056b790f90747f3a268edcd83510ca4ade4b5f6dcb8c31ed52a378c8a8b" exitCode=0 Oct 30 13:37:30.601110 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.601001 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr" Oct 30 13:37:30.601110 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.600997 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr" event={"ID":"2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc","Type":"ContainerDied","Data":"3a37f056b790f90747f3a268edcd83510ca4ade4b5f6dcb8c31ed52a378c8a8b"} Oct 30 13:37:30.601195 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.601135 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr" event={"ID":"2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc","Type":"ContainerDied","Data":"0d6722645bc392e8ed4e38be22c45e58c2f2bcf3ba9ee299c4902d97ac7154e3"} Oct 30 13:37:30.601195 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.601164 4931 scope.go:122] "RemoveContainer" containerID="3a37f056b790f90747f3a268edcd83510ca4ade4b5f6dcb8c31ed52a378c8a8b" Oct 30 13:37:30.614730 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.614704 4931 scope.go:122] "RemoveContainer" containerID="3a37f056b790f90747f3a268edcd83510ca4ade4b5f6dcb8c31ed52a378c8a8b" Oct 30 13:37:30.615042 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:37:30.615019 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a37f056b790f90747f3a268edcd83510ca4ade4b5f6dcb8c31ed52a378c8a8b\": container with ID starting with 3a37f056b790f90747f3a268edcd83510ca4ade4b5f6dcb8c31ed52a378c8a8b not found: ID does not exist" containerID="3a37f056b790f90747f3a268edcd83510ca4ade4b5f6dcb8c31ed52a378c8a8b" Oct 30 13:37:30.615125 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.615056 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a37f056b790f90747f3a268edcd83510ca4ade4b5f6dcb8c31ed52a378c8a8b"} err="failed to get container status \"3a37f056b790f90747f3a268edcd83510ca4ade4b5f6dcb8c31ed52a378c8a8b\": rpc error: code = NotFound desc = could not find container \"3a37f056b790f90747f3a268edcd83510ca4ade4b5f6dcb8c31ed52a378c8a8b\": container with ID starting with 3a37f056b790f90747f3a268edcd83510ca4ade4b5f6dcb8c31ed52a378c8a8b not found: ID does not exist" Oct 30 13:37:30.683992 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.683950 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f409e7f-6b24-410e-896c-a8a557298edf-maas-api-tls\") pod \"maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs\" (UID: \"7f409e7f-6b24-410e-896c-a8a557298edf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs" Oct 30 13:37:30.684175 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.684044 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f409e7f-6b24-410e-896c-a8a557298edf-metrics-certs\") pod \"maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs\" (UID: \"7f409e7f-6b24-410e-896c-a8a557298edf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs" Oct 30 13:37:30.684175 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.684078 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6x6s\" (UniqueName: \"kubernetes.io/projected/7f409e7f-6b24-410e-896c-a8a557298edf-kube-api-access-d6x6s\") pod \"maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs\" (UID: \"7f409e7f-6b24-410e-896c-a8a557298edf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs" Oct 30 13:37:30.684383 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.684363 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f409e7f-6b24-410e-896c-a8a557298edf-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs\" (UID: \"7f409e7f-6b24-410e-896c-a8a557298edf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs" Oct 30 13:37:30.686410 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.686381 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr"] Oct 30 13:37:30.687061 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.687034 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f409e7f-6b24-410e-896c-a8a557298edf-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs\" (UID: \"7f409e7f-6b24-410e-896c-a8a557298edf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs" Oct 30 13:37:30.687238 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.687222 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f409e7f-6b24-410e-896c-a8a557298edf-maas-api-tls\") pod \"maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs\" (UID: \"7f409e7f-6b24-410e-896c-a8a557298edf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs" Oct 30 13:37:30.687742 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.687723 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f409e7f-6b24-410e-896c-a8a557298edf-metrics-certs\") pod \"maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs\" (UID: \"7f409e7f-6b24-410e-896c-a8a557298edf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs" Oct 30 13:37:30.688963 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.688942 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03b5b3-5658b5f456-hwhrr"] Oct 30 13:37:30.761072 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.761032 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6x6s\" (UniqueName: \"kubernetes.io/projected/7f409e7f-6b24-410e-896c-a8a557298edf-kube-api-access-d6x6s\") pod \"maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs\" (UID: \"7f409e7f-6b24-410e-896c-a8a557298edf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs" Oct 30 13:37:30.816727 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:30.816689 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs" Oct 30 13:37:31.261812 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:37:31.261776 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f409e7f_6b24_410e_896c_a8a557298edf.slice/crio-91f42af699552dd2c69ade4dc10b9dc5aedfad8a6311dbe843c7774c6e2f92dd WatchSource:0}: Error finding container 91f42af699552dd2c69ade4dc10b9dc5aedfad8a6311dbe843c7774c6e2f92dd: Status 404 returned error can't find the container with id 91f42af699552dd2c69ade4dc10b9dc5aedfad8a6311dbe843c7774c6e2f92dd Oct 30 13:37:31.263792 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:31.263768 4931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:37:31.265176 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:31.265131 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs"] Oct 30 13:37:31.518151 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:31.518073 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc" path="/var/lib/kubelet/pods/2e5ba5bf-3b97-4afa-bc8c-09f151c6dfbc/volumes" Oct 30 13:37:31.577324 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:31.577286 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 13:37:31.577488 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:31.577344 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" podUID="e46deba0-1ef3-4b54-9821-d4fbf394c9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 13:37:31.610939 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:31.610909 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs" event={"ID":"7f409e7f-6b24-410e-896c-a8a557298edf","Type":"ContainerStarted","Data":"91f42af699552dd2c69ade4dc10b9dc5aedfad8a6311dbe843c7774c6e2f92dd"} Oct 30 13:37:32.577501 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:32.577464 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 13:37:32.578016 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:32.577980 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" podUID="e46deba0-1ef3-4b54-9821-d4fbf394c9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 13:37:32.621227 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:32.621186 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs" event={"ID":"7f409e7f-6b24-410e-896c-a8a557298edf","Type":"ContainerStarted","Data":"a689cf489f9c72ecb3333b4ac24a4eab30a69cfa7f3a8d80c9efd2d02f690988"} Oct 30 13:37:32.621445 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:32.621306 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs" Oct 30 13:37:32.679013 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:32.678955 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs" podStartSLOduration=173.338372589 podStartE2EDuration="2m53.678941191s" totalImagesPullingTime="340.568602ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:32.67750992 +0000 UTC m=+3151.281008386" watchObservedRunningTime="2026-10-30 13:37:32.678941191 +0000 UTC m=+3151.282439637" Oct 30 13:37:33.592292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:33.578799 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 13:37:33.592292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:33.578922 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" podUID="e46deba0-1ef3-4b54-9821-d4fbf394c9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 13:37:34.091218 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.090944 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" Oct 30 13:37:34.105629 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.105570 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" Oct 30 13:37:34.175099 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.175045 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-workload-socket" pod "cf83a441-04d6-46bc-a81d-1c47f8a68e8e" (UID: "cf83a441-04d6-46bc-a81d-1c47f8a68e8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:34.183217 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.175115 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-workload-socket\") pod \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " Oct 30 13:37:34.183217 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.175537 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9babe7d-51a7-4d00-ab49-76441dd6dec5-credential-socket" pod "a9babe7d-51a7-4d00-ab49-76441dd6dec5" (UID: "a9babe7d-51a7-4d00-ab49-76441dd6dec5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:34.183217 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.175260 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9babe7d-51a7-4d00-ab49-76441dd6dec5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9babe7d-51a7-4d00-ab49-76441dd6dec5-credential-socket\") pod \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " Oct 30 13:37:34.183217 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.176375 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-kube-api-access-f65vq\" (UniqueName: \"kubernetes.io/projected/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-kube-api-access-f65vq\") pod \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " Oct 30 13:37:34.183217 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.176439 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9babe7d-51a7-4d00-ab49-76441dd6dec5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9babe7d-51a7-4d00-ab49-76441dd6dec5-istio-podinfo\") pod \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " Oct 30 13:37:34.183217 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.176505 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9babe7d-51a7-4d00-ab49-76441dd6dec5-istio-token\" (UniqueName: \"kubernetes.io/projected/a9babe7d-51a7-4d00-ab49-76441dd6dec5-istio-token\") pod \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " Oct 30 13:37:34.183217 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.176553 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9babe7d-51a7-4d00-ab49-76441dd6dec5-kube-api-access-w4jgj\" (UniqueName: \"kubernetes.io/projected/a9babe7d-51a7-4d00-ab49-76441dd6dec5-kube-api-access-w4jgj\") pod \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " Oct 30 13:37:34.183217 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.176647 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-istio-data\") pod \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " Oct 30 13:37:34.183217 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.176718 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-istio-podinfo\") pod \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " Oct 30 13:37:34.183217 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.176761 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-istio-envoy\") pod \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " Oct 30 13:37:34.183217 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.176841 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9babe7d-51a7-4d00-ab49-76441dd6dec5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9babe7d-51a7-4d00-ab49-76441dd6dec5-istio-data\") pod \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " Oct 30 13:37:34.183217 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.176897 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9babe7d-51a7-4d00-ab49-76441dd6dec5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9babe7d-51a7-4d00-ab49-76441dd6dec5-workload-certs\") pod \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " Oct 30 13:37:34.183217 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.176925 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9babe7d-51a7-4d00-ab49-76441dd6dec5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9babe7d-51a7-4d00-ab49-76441dd6dec5-istiod-ca-cert\") pod \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " Oct 30 13:37:34.183217 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.176960 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-istiod-ca-cert\") pod \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " Oct 30 13:37:34.183217 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.177020 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-workload-certs\") pod \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " Oct 30 13:37:34.183217 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.177053 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9babe7d-51a7-4d00-ab49-76441dd6dec5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9babe7d-51a7-4d00-ab49-76441dd6dec5-istio-envoy\") pod \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " Oct 30 13:37:34.183217 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.177109 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-credential-socket\") pod \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " Oct 30 13:37:34.183217 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.177191 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-istio-token\" (UniqueName: \"kubernetes.io/projected/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-istio-token\") pod \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\" (UID: \"cf83a441-04d6-46bc-a81d-1c47f8a68e8e\") " Oct 30 13:37:34.183217 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.177242 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9babe7d-51a7-4d00-ab49-76441dd6dec5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9babe7d-51a7-4d00-ab49-76441dd6dec5-workload-socket\") pod \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\" (UID: \"a9babe7d-51a7-4d00-ab49-76441dd6dec5\") " Oct 30 13:37:34.183217 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.178182 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:34.183217 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.178211 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9babe7d-51a7-4d00-ab49-76441dd6dec5-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:34.183217 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.178180 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-istio-data" pod "cf83a441-04d6-46bc-a81d-1c47f8a68e8e" (UID: "cf83a441-04d6-46bc-a81d-1c47f8a68e8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:34.183217 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.178502 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-credential-socket" pod "cf83a441-04d6-46bc-a81d-1c47f8a68e8e" (UID: "cf83a441-04d6-46bc-a81d-1c47f8a68e8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:34.183217 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.178640 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-istiod-ca-cert" pod "cf83a441-04d6-46bc-a81d-1c47f8a68e8e" (UID: "cf83a441-04d6-46bc-a81d-1c47f8a68e8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:37:34.183217 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.183156 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-istio-envoy" pod "cf83a441-04d6-46bc-a81d-1c47f8a68e8e" (UID: "cf83a441-04d6-46bc-a81d-1c47f8a68e8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:34.184884 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.183340 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9babe7d-51a7-4d00-ab49-76441dd6dec5-istio-data" pod "a9babe7d-51a7-4d00-ab49-76441dd6dec5" (UID: "a9babe7d-51a7-4d00-ab49-76441dd6dec5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:34.184884 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.183533 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9babe7d-51a7-4d00-ab49-76441dd6dec5-workload-certs" pod "a9babe7d-51a7-4d00-ab49-76441dd6dec5" (UID: "a9babe7d-51a7-4d00-ab49-76441dd6dec5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:34.184884 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.183886 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-workload-certs" pod "cf83a441-04d6-46bc-a81d-1c47f8a68e8e" (UID: "cf83a441-04d6-46bc-a81d-1c47f8a68e8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:34.184884 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.184334 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9babe7d-51a7-4d00-ab49-76441dd6dec5-workload-socket" pod "a9babe7d-51a7-4d00-ab49-76441dd6dec5" (UID: "a9babe7d-51a7-4d00-ab49-76441dd6dec5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:34.185173 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.185148 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9babe7d-51a7-4d00-ab49-76441dd6dec5-istiod-ca-cert" pod "a9babe7d-51a7-4d00-ab49-76441dd6dec5" (UID: "a9babe7d-51a7-4d00-ab49-76441dd6dec5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:37:34.185959 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.185927 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9babe7d-51a7-4d00-ab49-76441dd6dec5-istio-envoy" pod "a9babe7d-51a7-4d00-ab49-76441dd6dec5" (UID: "a9babe7d-51a7-4d00-ab49-76441dd6dec5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:34.187747 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.187718 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9babe7d-51a7-4d00-ab49-76441dd6dec5-istio-token" pod "a9babe7d-51a7-4d00-ab49-76441dd6dec5" (UID: "a9babe7d-51a7-4d00-ab49-76441dd6dec5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:37:34.188327 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.188304 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-istio-token" pod "cf83a441-04d6-46bc-a81d-1c47f8a68e8e" (UID: "cf83a441-04d6-46bc-a81d-1c47f8a68e8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:37:34.188691 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.188666 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-kube-api-access-f65vq" pod "cf83a441-04d6-46bc-a81d-1c47f8a68e8e" (UID: "cf83a441-04d6-46bc-a81d-1c47f8a68e8e"). InnerVolumeSpecName "kube-api-access-f65vq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:37:34.188805 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.188786 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-istio-podinfo" pod "cf83a441-04d6-46bc-a81d-1c47f8a68e8e" (UID: "cf83a441-04d6-46bc-a81d-1c47f8a68e8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:37:34.188916 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.188897 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9babe7d-51a7-4d00-ab49-76441dd6dec5-istio-podinfo" pod "a9babe7d-51a7-4d00-ab49-76441dd6dec5" (UID: "a9babe7d-51a7-4d00-ab49-76441dd6dec5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:37:34.190996 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.190970 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9babe7d-51a7-4d00-ab49-76441dd6dec5-kube-api-access-w4jgj" pod "a9babe7d-51a7-4d00-ab49-76441dd6dec5" (UID: "a9babe7d-51a7-4d00-ab49-76441dd6dec5"). InnerVolumeSpecName "kube-api-access-w4jgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:37:34.194201 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.194179 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z"] Oct 30 13:37:34.194852 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.194837 4931 cpu_manager.go:414] "containerMap: removing container" podUID="a9babe7d-51a7-4d00-ab49-76441dd6dec5" containerName="istio-proxy" Oct 30 13:37:34.194897 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.194854 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9babe7d-51a7-4d00-ab49-76441dd6dec5" containerName="istio-proxy" Oct 30 13:37:34.194897 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.194875 4931 cpu_manager.go:414] "containerMap: removing container" podUID="cf83a441-04d6-46bc-a81d-1c47f8a68e8e" containerName="istio-proxy" Oct 30 13:37:34.194897 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.194881 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf83a441-04d6-46bc-a81d-1c47f8a68e8e" containerName="istio-proxy" Oct 30 13:37:34.195031 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.195018 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9babe7d-51a7-4d00-ab49-76441dd6dec5" containerName="istio-proxy" Oct 30 13:37:34.195031 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.195028 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf83a441-04d6-46bc-a81d-1c47f8a68e8e" containerName="istio-proxy" Oct 30 13:37:34.200961 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.200932 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" Oct 30 13:37:34.219920 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.218958 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-1c8130-openshift-default-dockercfg-drhsq\"" Oct 30 13:37:34.221214 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.221005 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z"] Oct 30 13:37:34.280042 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.276761 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm"] Oct 30 13:37:34.280042 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.279294 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9916ba3d-3bef-4338-aa2e-f5c27782188f-credential-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" Oct 30 13:37:34.280042 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.279329 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9916ba3d-3bef-4338-aa2e-f5c27782188f-istio-data\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" Oct 30 13:37:34.280042 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.279418 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9916ba3d-3bef-4338-aa2e-f5c27782188f-workload-certs\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" Oct 30 13:37:34.280042 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.279468 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7mcv\" (UniqueName: \"kubernetes.io/projected/9916ba3d-3bef-4338-aa2e-f5c27782188f-kube-api-access-b7mcv\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" Oct 30 13:37:34.280042 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.279512 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9916ba3d-3bef-4338-aa2e-f5c27782188f-istio-podinfo\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" Oct 30 13:37:34.280042 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.279544 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9916ba3d-3bef-4338-aa2e-f5c27782188f-istio-token\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" Oct 30 13:37:34.280042 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.279559 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9916ba3d-3bef-4338-aa2e-f5c27782188f-istiod-ca-cert\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" Oct 30 13:37:34.280042 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.279600 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9916ba3d-3bef-4338-aa2e-f5c27782188f-workload-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" Oct 30 13:37:34.280042 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.279656 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9916ba3d-3bef-4338-aa2e-f5c27782188f-istio-envoy\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" Oct 30 13:37:34.280042 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.279714 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f65vq\" (UniqueName: \"kubernetes.io/projected/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-kube-api-access-f65vq\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:34.280042 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.279726 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9babe7d-51a7-4d00-ab49-76441dd6dec5-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:34.280042 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.279736 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9babe7d-51a7-4d00-ab49-76441dd6dec5-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:34.280042 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.279745 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4jgj\" (UniqueName: \"kubernetes.io/projected/a9babe7d-51a7-4d00-ab49-76441dd6dec5-kube-api-access-w4jgj\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:34.280042 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.279754 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:34.280042 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.279763 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:34.280042 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.279772 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:34.280042 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.279780 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9babe7d-51a7-4d00-ab49-76441dd6dec5-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:34.280042 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.279788 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9babe7d-51a7-4d00-ab49-76441dd6dec5-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:34.280042 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.279797 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9babe7d-51a7-4d00-ab49-76441dd6dec5-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:34.280042 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.279805 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:34.280042 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.279813 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:34.280042 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.279822 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9babe7d-51a7-4d00-ab49-76441dd6dec5-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:34.280042 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.279829 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:34.280042 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.279838 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf83a441-04d6-46bc-a81d-1c47f8a68e8e-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:34.280042 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.279848 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9babe7d-51a7-4d00-ab49-76441dd6dec5-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:34.283249 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.283218 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" Oct 30 13:37:34.285879 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.285850 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm"] Oct 30 13:37:34.289678 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.289654 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" Oct 30 13:37:34.381211 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.381127 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d987afd-6dc8-401b-a339-3909dff59665-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d987afd-6dc8-401b-a339-3909dff59665-istiod-ca-cert\") pod \"6d987afd-6dc8-401b-a339-3909dff59665\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " Oct 30 13:37:34.381211 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.381195 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d987afd-6dc8-401b-a339-3909dff59665-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d987afd-6dc8-401b-a339-3909dff59665-workload-certs\") pod \"6d987afd-6dc8-401b-a339-3909dff59665\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " Oct 30 13:37:34.381457 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.381320 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d987afd-6dc8-401b-a339-3909dff59665-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d987afd-6dc8-401b-a339-3909dff59665-istio-envoy\") pod \"6d987afd-6dc8-401b-a339-3909dff59665\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " Oct 30 13:37:34.381457 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.381387 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d987afd-6dc8-401b-a339-3909dff59665-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d987afd-6dc8-401b-a339-3909dff59665-credential-socket\") pod \"6d987afd-6dc8-401b-a339-3909dff59665\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " Oct 30 13:37:34.381457 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.381424 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d987afd-6dc8-401b-a339-3909dff59665-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d987afd-6dc8-401b-a339-3909dff59665-istio-podinfo\") pod \"6d987afd-6dc8-401b-a339-3909dff59665\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " Oct 30 13:37:34.381657 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.381489 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d987afd-6dc8-401b-a339-3909dff59665-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d987afd-6dc8-401b-a339-3909dff59665-istio-data\") pod \"6d987afd-6dc8-401b-a339-3909dff59665\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " Oct 30 13:37:34.381657 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.381528 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d987afd-6dc8-401b-a339-3909dff59665-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d987afd-6dc8-401b-a339-3909dff59665-workload-socket\") pod \"6d987afd-6dc8-401b-a339-3909dff59665\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " Oct 30 13:37:34.381657 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.381552 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d987afd-6dc8-401b-a339-3909dff59665-kube-api-access-4ljbc\" (UniqueName: \"kubernetes.io/projected/6d987afd-6dc8-401b-a339-3909dff59665-kube-api-access-4ljbc\") pod \"6d987afd-6dc8-401b-a339-3909dff59665\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " Oct 30 13:37:34.381657 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.381596 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d987afd-6dc8-401b-a339-3909dff59665-istio-token\" (UniqueName: \"kubernetes.io/projected/6d987afd-6dc8-401b-a339-3909dff59665-istio-token\") pod \"6d987afd-6dc8-401b-a339-3909dff59665\" (UID: \"6d987afd-6dc8-401b-a339-3909dff59665\") " Oct 30 13:37:34.381657 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.381592 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d987afd-6dc8-401b-a339-3909dff59665-istiod-ca-cert" pod "6d987afd-6dc8-401b-a339-3909dff59665" (UID: "6d987afd-6dc8-401b-a339-3909dff59665"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:37:34.381935 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.381773 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-istio-envoy\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" Oct 30 13:37:34.381935 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.381808 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5b5p\" (UniqueName: \"kubernetes.io/projected/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-kube-api-access-d5b5p\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" Oct 30 13:37:34.381935 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.381851 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9916ba3d-3bef-4338-aa2e-f5c27782188f-credential-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" Oct 30 13:37:34.381935 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.381870 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9916ba3d-3bef-4338-aa2e-f5c27782188f-istio-data\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" Oct 30 13:37:34.381935 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.381932 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-istio-token\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" Oct 30 13:37:34.382224 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.381999 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-istio-data\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" Oct 30 13:37:34.382224 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.382039 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-credential-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" Oct 30 13:37:34.382224 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.382074 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9916ba3d-3bef-4338-aa2e-f5c27782188f-workload-certs\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" Oct 30 13:37:34.382224 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.382160 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7mcv\" (UniqueName: \"kubernetes.io/projected/9916ba3d-3bef-4338-aa2e-f5c27782188f-kube-api-access-b7mcv\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" Oct 30 13:37:34.382437 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.382231 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-workload-certs\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" Oct 30 13:37:34.382437 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.382304 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9916ba3d-3bef-4338-aa2e-f5c27782188f-istio-podinfo\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" Oct 30 13:37:34.382437 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.382368 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9916ba3d-3bef-4338-aa2e-f5c27782188f-istio-token\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" Oct 30 13:37:34.382437 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.382393 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9916ba3d-3bef-4338-aa2e-f5c27782188f-istiod-ca-cert\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" Oct 30 13:37:34.382921 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.382896 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9916ba3d-3bef-4338-aa2e-f5c27782188f-credential-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" Oct 30 13:37:34.383380 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.383328 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-istio-podinfo\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" Oct 30 13:37:34.383487 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.383414 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9916ba3d-3bef-4338-aa2e-f5c27782188f-workload-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" Oct 30 13:37:34.383564 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.383487 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-istiod-ca-cert\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" Oct 30 13:37:34.383564 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.383522 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-workload-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" Oct 30 13:37:34.383687 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.383590 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9916ba3d-3bef-4338-aa2e-f5c27782188f-istio-envoy\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" Oct 30 13:37:34.383753 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.383712 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d987afd-6dc8-401b-a339-3909dff59665-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:34.383855 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.383833 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9916ba3d-3bef-4338-aa2e-f5c27782188f-istio-data\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" Oct 30 13:37:34.384307 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.384288 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9916ba3d-3bef-4338-aa2e-f5c27782188f-workload-certs\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" Oct 30 13:37:34.384434 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.384381 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d987afd-6dc8-401b-a339-3909dff59665-istio-podinfo" pod "6d987afd-6dc8-401b-a339-3909dff59665" (UID: "6d987afd-6dc8-401b-a339-3909dff59665"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:37:34.384561 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.384504 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d987afd-6dc8-401b-a339-3909dff59665-istio-token" pod "6d987afd-6dc8-401b-a339-3909dff59665" (UID: "6d987afd-6dc8-401b-a339-3909dff59665"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:37:34.384647 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.384591 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d987afd-6dc8-401b-a339-3909dff59665-credential-socket" pod "6d987afd-6dc8-401b-a339-3909dff59665" (UID: "6d987afd-6dc8-401b-a339-3909dff59665"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:34.384783 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.384760 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d987afd-6dc8-401b-a339-3909dff59665-istio-data" pod "6d987afd-6dc8-401b-a339-3909dff59665" (UID: "6d987afd-6dc8-401b-a339-3909dff59665"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:34.385004 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.384964 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d987afd-6dc8-401b-a339-3909dff59665-workload-socket" pod "6d987afd-6dc8-401b-a339-3909dff59665" (UID: "6d987afd-6dc8-401b-a339-3909dff59665"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:34.385191 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.385166 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d987afd-6dc8-401b-a339-3909dff59665-istio-envoy" pod "6d987afd-6dc8-401b-a339-3909dff59665" (UID: "6d987afd-6dc8-401b-a339-3909dff59665"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:34.385451 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.385431 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d987afd-6dc8-401b-a339-3909dff59665-workload-certs" pod "6d987afd-6dc8-401b-a339-3909dff59665" (UID: "6d987afd-6dc8-401b-a339-3909dff59665"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:34.386006 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.385986 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9916ba3d-3bef-4338-aa2e-f5c27782188f-workload-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" Oct 30 13:37:34.386522 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.386494 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9916ba3d-3bef-4338-aa2e-f5c27782188f-istiod-ca-cert\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" Oct 30 13:37:34.386979 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.386956 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d987afd-6dc8-401b-a339-3909dff59665-kube-api-access-4ljbc" pod "6d987afd-6dc8-401b-a339-3909dff59665" (UID: "6d987afd-6dc8-401b-a339-3909dff59665"). InnerVolumeSpecName "kube-api-access-4ljbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:37:34.387412 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.387394 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9916ba3d-3bef-4338-aa2e-f5c27782188f-istio-envoy\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" Oct 30 13:37:34.389675 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.389652 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9916ba3d-3bef-4338-aa2e-f5c27782188f-istio-podinfo\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" Oct 30 13:37:34.411204 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.411132 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7mcv\" (UniqueName: \"kubernetes.io/projected/9916ba3d-3bef-4338-aa2e-f5c27782188f-kube-api-access-b7mcv\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" Oct 30 13:37:34.411204 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.411140 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9916ba3d-3bef-4338-aa2e-f5c27782188f-istio-token\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" Oct 30 13:37:34.468390 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.468350 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p"] Oct 30 13:37:34.469731 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.469701 4931 cpu_manager.go:414] "containerMap: removing container" podUID="6d987afd-6dc8-401b-a339-3909dff59665" containerName="istio-proxy" Oct 30 13:37:34.469956 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.469940 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d987afd-6dc8-401b-a339-3909dff59665" containerName="istio-proxy" Oct 30 13:37:34.470360 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.470341 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d987afd-6dc8-401b-a339-3909dff59665" containerName="istio-proxy" Oct 30 13:37:34.478391 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.478355 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p"] Oct 30 13:37:34.478529 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.478467 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" Oct 30 13:37:34.484884 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.484859 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-istio-token\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" Oct 30 13:37:34.485160 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.485136 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-istio-data\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" Oct 30 13:37:34.485258 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.485192 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-credential-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" Oct 30 13:37:34.485712 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.485686 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-istio-data\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" Oct 30 13:37:34.485857 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.485830 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-workload-certs\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" Oct 30 13:37:34.485950 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.485933 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-istio-podinfo\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" Oct 30 13:37:34.486036 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.486017 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-istiod-ca-cert\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" Oct 30 13:37:34.486093 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.486058 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-workload-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" Oct 30 13:37:34.486174 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.486158 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-istio-envoy\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" Oct 30 13:37:34.486231 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.486208 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5b5p\" (UniqueName: \"kubernetes.io/projected/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-kube-api-access-d5b5p\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" Oct 30 13:37:34.487236 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.486908 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d987afd-6dc8-401b-a339-3909dff59665-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:34.487377 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.487252 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d987afd-6dc8-401b-a339-3909dff59665-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:34.487377 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.487299 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d987afd-6dc8-401b-a339-3909dff59665-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:34.487377 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.487315 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d987afd-6dc8-401b-a339-3909dff59665-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:34.487377 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.487331 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ljbc\" (UniqueName: \"kubernetes.io/projected/6d987afd-6dc8-401b-a339-3909dff59665-kube-api-access-4ljbc\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:34.487377 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.487344 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d987afd-6dc8-401b-a339-3909dff59665-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:34.487377 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.487361 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d987afd-6dc8-401b-a339-3909dff59665-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:34.487377 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.487376 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d987afd-6dc8-401b-a339-3909dff59665-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:34.487764 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.487180 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-credential-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" Oct 30 13:37:34.487764 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.487670 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-workload-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" Oct 30 13:37:34.488316 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.488062 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-workload-certs\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" Oct 30 13:37:34.488407 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.486945 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-istiod-ca-cert\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" Oct 30 13:37:34.491676 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.490216 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-istio-envoy\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" Oct 30 13:37:34.491676 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.490455 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-istio-podinfo\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" Oct 30 13:37:34.512981 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.512949 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-istio-token\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" Oct 30 13:37:34.513308 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.513289 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5b5p\" (UniqueName: \"kubernetes.io/projected/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-kube-api-access-d5b5p\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" Oct 30 13:37:34.531601 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.531575 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" Oct 30 13:37:34.576972 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.576944 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 13:37:34.577079 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.576998 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" podUID="e46deba0-1ef3-4b54-9821-d4fbf394c9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 13:37:34.589112 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.589078 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6f71a7e-fbeb-4e16-8863-e17fc9695553-workload-certs\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" Oct 30 13:37:34.589309 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.589160 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6f71a7e-fbeb-4e16-8863-e17fc9695553-istio-podinfo\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" Oct 30 13:37:34.589309 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.589242 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6f71a7e-fbeb-4e16-8863-e17fc9695553-istio-data\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" Oct 30 13:37:34.589516 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.589337 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f71a7e-fbeb-4e16-8863-e17fc9695553-credential-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" Oct 30 13:37:34.589516 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.589387 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6f71a7e-fbeb-4e16-8863-e17fc9695553-istiod-ca-cert\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" Oct 30 13:37:34.589516 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.589452 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6f71a7e-fbeb-4e16-8863-e17fc9695553-istio-token\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" Oct 30 13:37:34.589516 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.589478 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl4qt\" (UniqueName: \"kubernetes.io/projected/d6f71a7e-fbeb-4e16-8863-e17fc9695553-kube-api-access-kl4qt\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" Oct 30 13:37:34.589516 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.589502 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f71a7e-fbeb-4e16-8863-e17fc9695553-workload-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" Oct 30 13:37:34.589770 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.589556 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6f71a7e-fbeb-4e16-8863-e17fc9695553-istio-envoy\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" Oct 30 13:37:34.606506 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.606025 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" Oct 30 13:37:34.688006 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.683956 4931 generic.go:356] "Generic (PLEG): container finished" podID="6d987afd-6dc8-401b-a339-3909dff59665" containerID="4bdcfb97e3fe87e7eea3b0d9d44cea2ede26f7e1db3835cf001a958f272cf289" exitCode=0 Oct 30 13:37:34.688006 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.684235 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" event={"ID":"6d987afd-6dc8-401b-a339-3909dff59665","Type":"ContainerDied","Data":"4bdcfb97e3fe87e7eea3b0d9d44cea2ede26f7e1db3835cf001a958f272cf289"} Oct 30 13:37:34.688006 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.684360 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" event={"ID":"6d987afd-6dc8-401b-a339-3909dff59665","Type":"ContainerDied","Data":"cbe618eff40eca857647eaad99b05b141a55c622118b490cc8c3446b095363bf"} Oct 30 13:37:34.688006 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.684408 4931 scope.go:122] "RemoveContainer" containerID="4bdcfb97e3fe87e7eea3b0d9d44cea2ede26f7e1db3835cf001a958f272cf289" Oct 30 13:37:34.688006 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.684497 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq" Oct 30 13:37:34.691973 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.691944 4931 generic.go:356] "Generic (PLEG): container finished" podID="cf83a441-04d6-46bc-a81d-1c47f8a68e8e" containerID="830343a1f2bbfbd0ee3a784909a82cacc2fd9005ffa15706b14d001645fb5816" exitCode=0 Oct 30 13:37:34.692336 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.692307 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" event={"ID":"cf83a441-04d6-46bc-a81d-1c47f8a68e8e","Type":"ContainerDied","Data":"830343a1f2bbfbd0ee3a784909a82cacc2fd9005ffa15706b14d001645fb5816"} Oct 30 13:37:34.692571 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.692537 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" event={"ID":"cf83a441-04d6-46bc-a81d-1c47f8a68e8e","Type":"ContainerDied","Data":"aaaff346151a2fea891db93a452c42bcd2145f8825c893ab615a729f93ebbd5e"} Oct 30 13:37:34.693288 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.693238 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht" Oct 30 13:37:34.695173 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.695154 4931 generic.go:356] "Generic (PLEG): container finished" podID="a9babe7d-51a7-4d00-ab49-76441dd6dec5" containerID="2e3198b0df152359f3a8fb485a11d9f4c9f2d5ecb996eaae31ee1d0b009db496" exitCode=0 Oct 30 13:37:34.695406 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.695384 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" event={"ID":"a9babe7d-51a7-4d00-ab49-76441dd6dec5","Type":"ContainerDied","Data":"2e3198b0df152359f3a8fb485a11d9f4c9f2d5ecb996eaae31ee1d0b009db496"} Oct 30 13:37:34.695604 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.695568 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" event={"ID":"a9babe7d-51a7-4d00-ab49-76441dd6dec5","Type":"ContainerDied","Data":"3919e3ee761378c8d2520974003323826c0085a2e44dd44c392ca0b29e44e818"} Oct 30 13:37:34.695892 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.695871 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7" Oct 30 13:37:34.701211 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.701177 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6f71a7e-fbeb-4e16-8863-e17fc9695553-workload-certs\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" Oct 30 13:37:34.701366 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.701325 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6f71a7e-fbeb-4e16-8863-e17fc9695553-istio-podinfo\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" Oct 30 13:37:34.702442 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.702417 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6f71a7e-fbeb-4e16-8863-e17fc9695553-workload-certs\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" Oct 30 13:37:34.703859 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.703826 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6f71a7e-fbeb-4e16-8863-e17fc9695553-istio-data\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" Oct 30 13:37:34.703962 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.703927 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f71a7e-fbeb-4e16-8863-e17fc9695553-credential-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" Oct 30 13:37:34.704355 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.703977 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6f71a7e-fbeb-4e16-8863-e17fc9695553-istiod-ca-cert\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" Oct 30 13:37:34.704451 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.704416 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6f71a7e-fbeb-4e16-8863-e17fc9695553-istio-token\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" Oct 30 13:37:34.704521 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.704454 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl4qt\" (UniqueName: \"kubernetes.io/projected/d6f71a7e-fbeb-4e16-8863-e17fc9695553-kube-api-access-kl4qt\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" Oct 30 13:37:34.704817 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.704790 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6f71a7e-fbeb-4e16-8863-e17fc9695553-istiod-ca-cert\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" Oct 30 13:37:34.705448 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.705420 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f71a7e-fbeb-4e16-8863-e17fc9695553-workload-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" Oct 30 13:37:34.706067 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.705712 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6f71a7e-fbeb-4e16-8863-e17fc9695553-istio-envoy\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" Oct 30 13:37:34.706067 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.705734 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6f71a7e-fbeb-4e16-8863-e17fc9695553-istio-data\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" Oct 30 13:37:34.706067 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.705813 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f71a7e-fbeb-4e16-8863-e17fc9695553-workload-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" Oct 30 13:37:34.706296 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.706119 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f71a7e-fbeb-4e16-8863-e17fc9695553-credential-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" Oct 30 13:37:34.710823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.710802 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6f71a7e-fbeb-4e16-8863-e17fc9695553-istio-envoy\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" Oct 30 13:37:34.711372 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.710819 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6f71a7e-fbeb-4e16-8863-e17fc9695553-istio-podinfo\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" Oct 30 13:37:34.719399 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.719375 4931 scope.go:122] "RemoveContainer" containerID="4bdcfb97e3fe87e7eea3b0d9d44cea2ede26f7e1db3835cf001a958f272cf289" Oct 30 13:37:34.719954 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:37:34.719930 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bdcfb97e3fe87e7eea3b0d9d44cea2ede26f7e1db3835cf001a958f272cf289\": container with ID starting with 4bdcfb97e3fe87e7eea3b0d9d44cea2ede26f7e1db3835cf001a958f272cf289 not found: ID does not exist" containerID="4bdcfb97e3fe87e7eea3b0d9d44cea2ede26f7e1db3835cf001a958f272cf289" Oct 30 13:37:34.720230 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.720202 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bdcfb97e3fe87e7eea3b0d9d44cea2ede26f7e1db3835cf001a958f272cf289"} err="failed to get container status \"4bdcfb97e3fe87e7eea3b0d9d44cea2ede26f7e1db3835cf001a958f272cf289\": rpc error: code = NotFound desc = could not find container \"4bdcfb97e3fe87e7eea3b0d9d44cea2ede26f7e1db3835cf001a958f272cf289\": container with ID starting with 4bdcfb97e3fe87e7eea3b0d9d44cea2ede26f7e1db3835cf001a958f272cf289 not found: ID does not exist" Oct 30 13:37:34.720822 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.720805 4931 scope.go:122] "RemoveContainer" containerID="830343a1f2bbfbd0ee3a784909a82cacc2fd9005ffa15706b14d001645fb5816" Oct 30 13:37:34.766456 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.766385 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl4qt\" (UniqueName: \"kubernetes.io/projected/d6f71a7e-fbeb-4e16-8863-e17fc9695553-kube-api-access-kl4qt\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" Oct 30 13:37:34.775909 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.775875 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6f71a7e-fbeb-4e16-8863-e17fc9695553-istio-token\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" Oct 30 13:37:34.796591 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.796559 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" Oct 30 13:37:34.807952 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.802325 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq"] Oct 30 13:37:34.809359 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.809329 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq"] Oct 30 13:37:34.865290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.858747 4931 scope.go:122] "RemoveContainer" containerID="830343a1f2bbfbd0ee3a784909a82cacc2fd9005ffa15706b14d001645fb5816" Oct 30 13:37:34.866210 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:37:34.866029 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"830343a1f2bbfbd0ee3a784909a82cacc2fd9005ffa15706b14d001645fb5816\": container with ID starting with 830343a1f2bbfbd0ee3a784909a82cacc2fd9005ffa15706b14d001645fb5816 not found: ID does not exist" containerID="830343a1f2bbfbd0ee3a784909a82cacc2fd9005ffa15706b14d001645fb5816" Oct 30 13:37:34.866210 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.866074 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"830343a1f2bbfbd0ee3a784909a82cacc2fd9005ffa15706b14d001645fb5816"} err="failed to get container status \"830343a1f2bbfbd0ee3a784909a82cacc2fd9005ffa15706b14d001645fb5816\": rpc error: code = NotFound desc = could not find container \"830343a1f2bbfbd0ee3a784909a82cacc2fd9005ffa15706b14d001645fb5816\": container with ID starting with 830343a1f2bbfbd0ee3a784909a82cacc2fd9005ffa15706b14d001645fb5816 not found: ID does not exist" Oct 30 13:37:34.866210 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.866101 4931 scope.go:122] "RemoveContainer" containerID="2e3198b0df152359f3a8fb485a11d9f4c9f2d5ecb996eaae31ee1d0b009db496" Oct 30 13:37:34.886684 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.886655 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7"] Oct 30 13:37:34.891256 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.891229 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7"] Oct 30 13:37:34.900213 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:37:34.900161 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9916ba3d_3bef_4338_aa2e_f5c27782188f.slice/crio-6960c197e37643a0841ba0f3dcd2572a2eff526e231eb18a20eef476ed494d2b WatchSource:0}: Error finding container 6960c197e37643a0841ba0f3dcd2572a2eff526e231eb18a20eef476ed494d2b: Status 404 returned error can't find the container with id 6960c197e37643a0841ba0f3dcd2572a2eff526e231eb18a20eef476ed494d2b Oct 30 13:37:34.900754 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.900730 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z"] Oct 30 13:37:34.905390 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.905372 4931 scope.go:122] "RemoveContainer" containerID="2e3198b0df152359f3a8fb485a11d9f4c9f2d5ecb996eaae31ee1d0b009db496" Oct 30 13:37:34.905722 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:37:34.905658 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e3198b0df152359f3a8fb485a11d9f4c9f2d5ecb996eaae31ee1d0b009db496\": container with ID starting with 2e3198b0df152359f3a8fb485a11d9f4c9f2d5ecb996eaae31ee1d0b009db496 not found: ID does not exist" containerID="2e3198b0df152359f3a8fb485a11d9f4c9f2d5ecb996eaae31ee1d0b009db496" Oct 30 13:37:34.905793 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.905736 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e3198b0df152359f3a8fb485a11d9f4c9f2d5ecb996eaae31ee1d0b009db496"} err="failed to get container status \"2e3198b0df152359f3a8fb485a11d9f4c9f2d5ecb996eaae31ee1d0b009db496\": rpc error: code = NotFound desc = could not find container \"2e3198b0df152359f3a8fb485a11d9f4c9f2d5ecb996eaae31ee1d0b009db496\": container with ID starting with 2e3198b0df152359f3a8fb485a11d9f4c9f2d5ecb996eaae31ee1d0b009db496 not found: ID does not exist" Oct 30 13:37:34.906303 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.906258 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:37:34.906368 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.906354 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:37:34.906401 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.906392 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:37:34.942354 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.942320 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht"] Oct 30 13:37:34.971162 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.971112 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht"] Oct 30 13:37:34.985352 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.985309 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm"] Oct 30 13:37:34.992015 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:37:34.991888 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod813ffc2b_3c94_4847_aa9b_2d7971da6e4d.slice/crio-7710b912d480da97e9a2d1990ef54ba55ea540211f73e8c14c64f6f8af02df1b WatchSource:0}: Error finding container 7710b912d480da97e9a2d1990ef54ba55ea540211f73e8c14c64f6f8af02df1b: Status 404 returned error can't find the container with id 7710b912d480da97e9a2d1990ef54ba55ea540211f73e8c14c64f6f8af02df1b Oct 30 13:37:34.996153 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.996114 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:37:34.996283 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.996209 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:37:34.996283 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:34.996258 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:37:35.212891 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:35.212859 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p"] Oct 30 13:37:35.213112 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:37:35.212941 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f71a7e_fbeb_4e16_8863_e17fc9695553.slice/crio-9c4548d36d2c6c3db8004a50a3443d3f36e33dd15ceef2e3ada0c7fdf211d154 WatchSource:0}: Error finding container 9c4548d36d2c6c3db8004a50a3443d3f36e33dd15ceef2e3ada0c7fdf211d154: Status 404 returned error can't find the container with id 9c4548d36d2c6c3db8004a50a3443d3f36e33dd15ceef2e3ada0c7fdf211d154 Oct 30 13:37:35.217927 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:35.217874 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:37:35.218028 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:35.217986 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:37:35.218083 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:35.218037 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:37:35.522884 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:35.522844 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d987afd-6dc8-401b-a339-3909dff59665" path="/var/lib/kubelet/pods/6d987afd-6dc8-401b-a339-3909dff59665/volumes" Oct 30 13:37:35.523770 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:35.523716 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9babe7d-51a7-4d00-ab49-76441dd6dec5" path="/var/lib/kubelet/pods/a9babe7d-51a7-4d00-ab49-76441dd6dec5/volumes" Oct 30 13:37:35.524526 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:35.524505 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf83a441-04d6-46bc-a81d-1c47f8a68e8e" path="/var/lib/kubelet/pods/cf83a441-04d6-46bc-a81d-1c47f8a68e8e/volumes" Oct 30 13:37:35.583034 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:35.582976 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 13:37:35.583327 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:35.583291 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" podUID="e46deba0-1ef3-4b54-9821-d4fbf394c9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 13:37:35.837342 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:35.822597 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" event={"ID":"813ffc2b-3c94-4847-aa9b-2d7971da6e4d","Type":"ContainerStarted","Data":"5498caf666415906672c692ac63dd600423c82d515682a426064d18b60c9fcb7"} Oct 30 13:37:35.837342 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:35.822671 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" event={"ID":"813ffc2b-3c94-4847-aa9b-2d7971da6e4d","Type":"ContainerStarted","Data":"7710b912d480da97e9a2d1990ef54ba55ea540211f73e8c14c64f6f8af02df1b"} Oct 30 13:37:35.852588 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:35.852544 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" event={"ID":"d6f71a7e-fbeb-4e16-8863-e17fc9695553","Type":"ContainerStarted","Data":"0ab016e7e6e72b08b3a2959061f4d839b39b1e4db4d5db283b4375e744948a45"} Oct 30 13:37:35.852862 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:35.852835 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" event={"ID":"d6f71a7e-fbeb-4e16-8863-e17fc9695553","Type":"ContainerStarted","Data":"9c4548d36d2c6c3db8004a50a3443d3f36e33dd15ceef2e3ada0c7fdf211d154"} Oct 30 13:37:35.871074 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:35.871028 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" event={"ID":"9916ba3d-3bef-4338-aa2e-f5c27782188f","Type":"ContainerStarted","Data":"84b32dc6f16656f85128e413f1bc9741ef177676729f2f0ba6c25a4ff550df82"} Oct 30 13:37:35.871262 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:35.871106 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" event={"ID":"9916ba3d-3bef-4338-aa2e-f5c27782188f","Type":"ContainerStarted","Data":"6960c197e37643a0841ba0f3dcd2572a2eff526e231eb18a20eef476ed494d2b"} Oct 30 13:37:35.880968 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:35.880869 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" podStartSLOduration=170.880849764 podStartE2EDuration="2m50.880849764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:35.876391899 +0000 UTC m=+3154.479890347" watchObservedRunningTime="2026-10-30 13:37:35.880849764 +0000 UTC m=+3154.484348211" Oct 30 13:37:35.927782 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:35.927700 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" podStartSLOduration=215.927678729 podStartE2EDuration="3m35.927678729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:35.925028559 +0000 UTC m=+3154.528527047" watchObservedRunningTime="2026-10-30 13:37:35.927678729 +0000 UTC m=+3154.531177197" Oct 30 13:37:35.987215 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:35.987140 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" podStartSLOduration=170.987116181 podStartE2EDuration="2m50.987116181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:35.980210137 +0000 UTC m=+3154.583708616" watchObservedRunningTime="2026-10-30 13:37:35.987116181 +0000 UTC m=+3154.590614630" Oct 30 13:37:36.542295 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:36.536504 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" Oct 30 13:37:36.555546 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:36.555508 4931 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 13:37:36.555797 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:36.555768 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" podUID="9916ba3d-3bef-4338-aa2e-f5c27782188f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 13:37:36.647293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:36.623029 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" Oct 30 13:37:36.649609 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:36.649581 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" Oct 30 13:37:36.710291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:36.702393 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" Oct 30 13:37:36.710818 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:36.710790 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" Oct 30 13:37:36.711102 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:36.711081 4931 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 30 13:37:36.711299 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:36.711251 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" podUID="813ffc2b-3c94-4847-aa9b-2d7971da6e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 30 13:37:36.824446 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:36.813678 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" Oct 30 13:37:36.824446 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:36.820973 4931 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 13:37:36.824446 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:36.821027 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" podUID="d6f71a7e-fbeb-4e16-8863-e17fc9695553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 13:37:37.488498 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:37:37.488459 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:37:37.610064 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:37.610028 4931 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 30 13:37:37.617394 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:37.610731 4931 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 13:37:37.617635 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:37.617609 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" podUID="9916ba3d-3bef-4338-aa2e-f5c27782188f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 13:37:37.617766 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:37.617352 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" podUID="813ffc2b-3c94-4847-aa9b-2d7971da6e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 30 13:37:37.800944 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:37.800838 4931 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 13:37:37.801246 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:37.801201 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" podUID="d6f71a7e-fbeb-4e16-8863-e17fc9695553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 13:37:38.549334 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:38.539382 4931 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 13:37:38.549334 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:38.539449 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" podUID="9916ba3d-3bef-4338-aa2e-f5c27782188f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 13:37:38.735086 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:38.733156 4931 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 30 13:37:38.735086 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:38.733225 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" podUID="813ffc2b-3c94-4847-aa9b-2d7971da6e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 30 13:37:38.780140 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:38.780097 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs" Oct 30 13:37:38.795305 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:38.793335 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4"] Oct 30 13:37:38.795305 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:38.793632 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" podUID="2776ed01-5f18-493e-bf9b-7fbb328e01a2" containerName="istio-proxy" containerID="cri-o://abe9d4324eceb24becf64e846066d1bd10981698ac0970ab00f5089dd831a6e5" gracePeriod=30 Oct 30 13:37:38.871494 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:38.867076 4931 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 13:37:38.871494 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:38.867135 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" podUID="d6f71a7e-fbeb-4e16-8863-e17fc9695553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 13:37:38.938314 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:38.936819 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7"] Oct 30 13:37:39.005316 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:38.993326 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79"] Oct 30 13:37:39.005316 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:38.993610 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" podUID="8bc1a121-7dc1-4ccd-a395-68929a6f6ad1" containerName="istio-proxy" containerID="cri-o://233b07ef31af5a34f76e1a82d0b290c0ff3ca24a2a1093c0b05c79e175c5c1bd" gracePeriod=30 Oct 30 13:37:39.225323 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:39.216150 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" podUID="550e0c64-4028-4fb3-98ef-fdc52cc66189" containerName="istio-proxy" containerID="cri-o://8c3e9c17377496135ef0edd2b29975097898c4bb897436fdd62c038eab997056" gracePeriod=30 Oct 30 13:37:39.564427 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:39.552139 4931 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 13:37:39.564427 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:39.552207 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" podUID="9916ba3d-3bef-4338-aa2e-f5c27782188f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 13:37:39.644289 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:39.639439 4931 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 30 13:37:39.644289 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:39.639515 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" podUID="813ffc2b-3c94-4847-aa9b-2d7971da6e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 30 13:37:39.666289 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:39.665311 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" podUID="65799c14-8de4-4b4a-90df-7cd257e1e581" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:37:39.692297 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:39.687376 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" podUID="ee8505de-cac0-4e49-bd15-a044536b10e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:37:39.825131 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:39.825032 4931 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 13:37:39.825421 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:39.825387 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" podUID="d6f71a7e-fbeb-4e16-8863-e17fc9695553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 13:37:39.876104 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:39.874934 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" podUID="606b252b-881c-4d40-8ba9-fc4ed937a995" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:37:39.962186 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:39.903758 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" podUID="64468005-61ea-4f2a-b72c-9c30c6a4d686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:37:39.973799 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:39.971907 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" podUID="9577f571-30cd-4043-a3e4-72bf95837552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:37:40.536292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:40.533289 4931 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 13:37:40.536292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:40.533351 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" podUID="9916ba3d-3bef-4338-aa2e-f5c27782188f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 13:37:40.616397 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:40.616358 4931 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 30 13:37:40.616814 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:40.616412 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" podUID="813ffc2b-3c94-4847-aa9b-2d7971da6e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 30 13:37:40.811351 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:40.810904 4931 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 13:37:40.811351 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:40.810969 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" podUID="d6f71a7e-fbeb-4e16-8863-e17fc9695553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 13:37:41.550302 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:41.547859 4931 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 13:37:41.550302 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:41.547921 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" podUID="9916ba3d-3bef-4338-aa2e-f5c27782188f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 13:37:41.618657 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:41.618620 4931 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 30 13:37:41.621907 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:41.621862 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" podUID="813ffc2b-3c94-4847-aa9b-2d7971da6e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 30 13:37:41.818345 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:41.800663 4931 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 13:37:41.820399 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:41.820351 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" podUID="d6f71a7e-fbeb-4e16-8863-e17fc9695553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 13:37:42.532199 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:42.532156 4931 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 13:37:42.532427 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:42.532216 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" podUID="9916ba3d-3bef-4338-aa2e-f5c27782188f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 13:37:42.606955 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:42.606916 4931 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 30 13:37:42.607233 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:42.607203 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" podUID="813ffc2b-3c94-4847-aa9b-2d7971da6e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 30 13:37:42.811641 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:42.811557 4931 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 13:37:42.812289 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:42.812235 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" podUID="d6f71a7e-fbeb-4e16-8863-e17fc9695553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 13:37:43.537079 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:43.537047 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" Oct 30 13:37:43.537553 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:43.537525 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" Oct 30 13:37:43.538374 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:43.538349 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" Oct 30 13:37:43.612493 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:43.612465 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" Oct 30 13:37:43.612765 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:43.612744 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" Oct 30 13:37:43.614574 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:43.614551 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" Oct 30 13:37:43.802712 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:43.802633 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" Oct 30 13:37:43.803129 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:43.803110 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" Oct 30 13:37:43.803850 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:43.803825 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" Oct 30 13:37:44.359187 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.359157 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2"] Oct 30 13:37:44.438124 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.436084 4931 generic.go:356] "Generic (PLEG): container finished" podID="2776ed01-5f18-493e-bf9b-7fbb328e01a2" containerID="abe9d4324eceb24becf64e846066d1bd10981698ac0970ab00f5089dd831a6e5" exitCode=0 Oct 30 13:37:44.438124 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.436312 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" event={"ID":"2776ed01-5f18-493e-bf9b-7fbb328e01a2","Type":"ContainerDied","Data":"abe9d4324eceb24becf64e846066d1bd10981698ac0970ab00f5089dd831a6e5"} Oct 30 13:37:44.457319 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.455856 4931 generic.go:356] "Generic (PLEG): container finished" podID="8bc1a121-7dc1-4ccd-a395-68929a6f6ad1" containerID="233b07ef31af5a34f76e1a82d0b290c0ff3ca24a2a1093c0b05c79e175c5c1bd" exitCode=0 Oct 30 13:37:44.457614 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.457555 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" event={"ID":"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1","Type":"ContainerDied","Data":"233b07ef31af5a34f76e1a82d0b290c0ff3ca24a2a1093c0b05c79e175c5c1bd"} Oct 30 13:37:44.774434 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.771502 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" Oct 30 13:37:44.811340 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.811216 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" Oct 30 13:37:44.871161 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.871128 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-kube-api-access-6ksjz\" (UniqueName: \"kubernetes.io/projected/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-kube-api-access-6ksjz\") pod \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " Oct 30 13:37:44.871161 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.871168 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-istio-token\" (UniqueName: \"kubernetes.io/projected/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-istio-token\") pod \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " Oct 30 13:37:44.871459 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.871205 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-istio-envoy\") pod \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " Oct 30 13:37:44.871459 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.871247 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-workload-certs\") pod \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " Oct 30 13:37:44.871583 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.871477 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-workload-socket\") pod \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " Oct 30 13:37:44.871583 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.871559 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-istio-podinfo\") pod \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " Oct 30 13:37:44.871702 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.871683 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-istio-data\") pod \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " Oct 30 13:37:44.871809 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.871793 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-istiod-ca-cert\") pod \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " Oct 30 13:37:44.871869 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.871860 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-credential-socket\") pod \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\" (UID: \"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1\") " Oct 30 13:37:44.872084 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.872062 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-workload-certs" pod "8bc1a121-7dc1-4ccd-a395-68929a6f6ad1" (UID: "8bc1a121-7dc1-4ccd-a395-68929a6f6ad1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:44.872566 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.872541 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-istiod-ca-cert" pod "8bc1a121-7dc1-4ccd-a395-68929a6f6ad1" (UID: "8bc1a121-7dc1-4ccd-a395-68929a6f6ad1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:37:44.872786 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.872659 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-workload-socket" pod "8bc1a121-7dc1-4ccd-a395-68929a6f6ad1" (UID: "8bc1a121-7dc1-4ccd-a395-68929a6f6ad1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:44.872786 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.872750 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-credential-socket" pod "8bc1a121-7dc1-4ccd-a395-68929a6f6ad1" (UID: "8bc1a121-7dc1-4ccd-a395-68929a6f6ad1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:44.872915 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.872835 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:44.872915 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.872850 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:44.872915 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.872860 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:44.873080 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.873021 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-istio-data" pod "8bc1a121-7dc1-4ccd-a395-68929a6f6ad1" (UID: "8bc1a121-7dc1-4ccd-a395-68929a6f6ad1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:44.874304 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.874262 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-kube-api-access-6ksjz" pod "8bc1a121-7dc1-4ccd-a395-68929a6f6ad1" (UID: "8bc1a121-7dc1-4ccd-a395-68929a6f6ad1"). InnerVolumeSpecName "kube-api-access-6ksjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:37:44.880002 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.879961 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-istio-envoy" pod "8bc1a121-7dc1-4ccd-a395-68929a6f6ad1" (UID: "8bc1a121-7dc1-4ccd-a395-68929a6f6ad1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:44.880468 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.880424 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-istio-podinfo" pod "8bc1a121-7dc1-4ccd-a395-68929a6f6ad1" (UID: "8bc1a121-7dc1-4ccd-a395-68929a6f6ad1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:37:44.882575 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.882551 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-istio-token" pod "8bc1a121-7dc1-4ccd-a395-68929a6f6ad1" (UID: "8bc1a121-7dc1-4ccd-a395-68929a6f6ad1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:37:44.919609 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.919580 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" Oct 30 13:37:44.974338 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.974294 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/550e0c64-4028-4fb3-98ef-fdc52cc66189-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/550e0c64-4028-4fb3-98ef-fdc52cc66189-credential-socket\") pod \"550e0c64-4028-4fb3-98ef-fdc52cc66189\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " Oct 30 13:37:44.974560 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.974394 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/550e0c64-4028-4fb3-98ef-fdc52cc66189-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/550e0c64-4028-4fb3-98ef-fdc52cc66189-workload-certs\") pod \"550e0c64-4028-4fb3-98ef-fdc52cc66189\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " Oct 30 13:37:44.974560 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.974483 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/550e0c64-4028-4fb3-98ef-fdc52cc66189-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/550e0c64-4028-4fb3-98ef-fdc52cc66189-istiod-ca-cert\") pod \"550e0c64-4028-4fb3-98ef-fdc52cc66189\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " Oct 30 13:37:44.974560 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.974549 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/550e0c64-4028-4fb3-98ef-fdc52cc66189-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/550e0c64-4028-4fb3-98ef-fdc52cc66189-istio-podinfo\") pod \"550e0c64-4028-4fb3-98ef-fdc52cc66189\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " Oct 30 13:37:44.974748 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.974608 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/550e0c64-4028-4fb3-98ef-fdc52cc66189-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/550e0c64-4028-4fb3-98ef-fdc52cc66189-workload-socket\") pod \"550e0c64-4028-4fb3-98ef-fdc52cc66189\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " Oct 30 13:37:44.974748 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.974672 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/550e0c64-4028-4fb3-98ef-fdc52cc66189-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/550e0c64-4028-4fb3-98ef-fdc52cc66189-istio-envoy\") pod \"550e0c64-4028-4fb3-98ef-fdc52cc66189\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " Oct 30 13:37:44.974926 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.974796 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/550e0c64-4028-4fb3-98ef-fdc52cc66189-workload-certs" pod "550e0c64-4028-4fb3-98ef-fdc52cc66189" (UID: "550e0c64-4028-4fb3-98ef-fdc52cc66189"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:44.974926 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.974840 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/550e0c64-4028-4fb3-98ef-fdc52cc66189-istio-token\" (UniqueName: \"kubernetes.io/projected/550e0c64-4028-4fb3-98ef-fdc52cc66189-istio-token\") pod \"550e0c64-4028-4fb3-98ef-fdc52cc66189\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " Oct 30 13:37:44.975211 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.975106 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/550e0c64-4028-4fb3-98ef-fdc52cc66189-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/550e0c64-4028-4fb3-98ef-fdc52cc66189-istio-data\") pod \"550e0c64-4028-4fb3-98ef-fdc52cc66189\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " Oct 30 13:37:44.975409 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.974789 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/550e0c64-4028-4fb3-98ef-fdc52cc66189-credential-socket" pod "550e0c64-4028-4fb3-98ef-fdc52cc66189" (UID: "550e0c64-4028-4fb3-98ef-fdc52cc66189"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:44.975552 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.975504 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/550e0c64-4028-4fb3-98ef-fdc52cc66189-workload-socket" pod "550e0c64-4028-4fb3-98ef-fdc52cc66189" (UID: "550e0c64-4028-4fb3-98ef-fdc52cc66189"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:44.975552 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.975423 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/550e0c64-4028-4fb3-98ef-fdc52cc66189-istiod-ca-cert" pod "550e0c64-4028-4fb3-98ef-fdc52cc66189" (UID: "550e0c64-4028-4fb3-98ef-fdc52cc66189"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:37:44.975813 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.975580 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/550e0c64-4028-4fb3-98ef-fdc52cc66189-kube-api-access-nlprd\" (UniqueName: \"kubernetes.io/projected/550e0c64-4028-4fb3-98ef-fdc52cc66189-kube-api-access-nlprd\") pod \"550e0c64-4028-4fb3-98ef-fdc52cc66189\" (UID: \"550e0c64-4028-4fb3-98ef-fdc52cc66189\") " Oct 30 13:37:44.975896 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.975844 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/550e0c64-4028-4fb3-98ef-fdc52cc66189-istio-data" pod "550e0c64-4028-4fb3-98ef-fdc52cc66189" (UID: "550e0c64-4028-4fb3-98ef-fdc52cc66189"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:44.976508 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.976474 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/550e0c64-4028-4fb3-98ef-fdc52cc66189-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:44.976508 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.976505 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:44.976754 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.976520 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ksjz\" (UniqueName: \"kubernetes.io/projected/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-kube-api-access-6ksjz\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:44.976754 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.976534 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:44.976754 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.976547 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:44.976754 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.976559 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/550e0c64-4028-4fb3-98ef-fdc52cc66189-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:44.976754 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.976571 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/550e0c64-4028-4fb3-98ef-fdc52cc66189-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:44.976754 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.976585 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:44.976754 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.976598 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/550e0c64-4028-4fb3-98ef-fdc52cc66189-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:44.976754 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.976612 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/550e0c64-4028-4fb3-98ef-fdc52cc66189-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:44.976754 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.976625 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:44.977645 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.977604 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/550e0c64-4028-4fb3-98ef-fdc52cc66189-istio-podinfo" pod "550e0c64-4028-4fb3-98ef-fdc52cc66189" (UID: "550e0c64-4028-4fb3-98ef-fdc52cc66189"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:37:44.978531 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.978492 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/550e0c64-4028-4fb3-98ef-fdc52cc66189-istio-envoy" pod "550e0c64-4028-4fb3-98ef-fdc52cc66189" (UID: "550e0c64-4028-4fb3-98ef-fdc52cc66189"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:44.978671 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.978646 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/550e0c64-4028-4fb3-98ef-fdc52cc66189-istio-token" pod "550e0c64-4028-4fb3-98ef-fdc52cc66189" (UID: "550e0c64-4028-4fb3-98ef-fdc52cc66189"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:37:44.979514 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:44.979483 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/550e0c64-4028-4fb3-98ef-fdc52cc66189-kube-api-access-nlprd" pod "550e0c64-4028-4fb3-98ef-fdc52cc66189" (UID: "550e0c64-4028-4fb3-98ef-fdc52cc66189"). InnerVolumeSpecName "kube-api-access-nlprd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:37:45.077665 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.077583 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2776ed01-5f18-493e-bf9b-7fbb328e01a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2776ed01-5f18-493e-bf9b-7fbb328e01a2-istiod-ca-cert\") pod \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " Oct 30 13:37:45.077665 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.077629 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2776ed01-5f18-493e-bf9b-7fbb328e01a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2776ed01-5f18-493e-bf9b-7fbb328e01a2-istio-podinfo\") pod \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " Oct 30 13:37:45.077665 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.077653 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2776ed01-5f18-493e-bf9b-7fbb328e01a2-istio-token\" (UniqueName: \"kubernetes.io/projected/2776ed01-5f18-493e-bf9b-7fbb328e01a2-istio-token\") pod \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " Oct 30 13:37:45.077966 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.077684 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2776ed01-5f18-493e-bf9b-7fbb328e01a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2776ed01-5f18-493e-bf9b-7fbb328e01a2-workload-socket\") pod \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " Oct 30 13:37:45.077966 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.077842 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2776ed01-5f18-493e-bf9b-7fbb328e01a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2776ed01-5f18-493e-bf9b-7fbb328e01a2-istio-data\") pod \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " Oct 30 13:37:45.077966 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.077903 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2776ed01-5f18-493e-bf9b-7fbb328e01a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2776ed01-5f18-493e-bf9b-7fbb328e01a2-credential-socket\") pod \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " Oct 30 13:37:45.078133 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.078023 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2776ed01-5f18-493e-bf9b-7fbb328e01a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2776ed01-5f18-493e-bf9b-7fbb328e01a2-workload-certs\") pod \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " Oct 30 13:37:45.078133 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.078064 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2776ed01-5f18-493e-bf9b-7fbb328e01a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2776ed01-5f18-493e-bf9b-7fbb328e01a2-istio-envoy\") pod \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " Oct 30 13:37:45.078232 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.078143 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2776ed01-5f18-493e-bf9b-7fbb328e01a2-istiod-ca-cert" pod "2776ed01-5f18-493e-bf9b-7fbb328e01a2" (UID: "2776ed01-5f18-493e-bf9b-7fbb328e01a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:37:45.078232 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.078160 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2776ed01-5f18-493e-bf9b-7fbb328e01a2-kube-api-access-cxfmv\" (UniqueName: \"kubernetes.io/projected/2776ed01-5f18-493e-bf9b-7fbb328e01a2-kube-api-access-cxfmv\") pod \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\" (UID: \"2776ed01-5f18-493e-bf9b-7fbb328e01a2\") " Oct 30 13:37:45.078232 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.078124 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2776ed01-5f18-493e-bf9b-7fbb328e01a2-istio-data" pod "2776ed01-5f18-493e-bf9b-7fbb328e01a2" (UID: "2776ed01-5f18-493e-bf9b-7fbb328e01a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:45.078232 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.078136 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2776ed01-5f18-493e-bf9b-7fbb328e01a2-workload-socket" pod "2776ed01-5f18-493e-bf9b-7fbb328e01a2" (UID: "2776ed01-5f18-493e-bf9b-7fbb328e01a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:45.078232 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.078196 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2776ed01-5f18-493e-bf9b-7fbb328e01a2-credential-socket" pod "2776ed01-5f18-493e-bf9b-7fbb328e01a2" (UID: "2776ed01-5f18-493e-bf9b-7fbb328e01a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:45.078529 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.078341 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2776ed01-5f18-493e-bf9b-7fbb328e01a2-workload-certs" pod "2776ed01-5f18-493e-bf9b-7fbb328e01a2" (UID: "2776ed01-5f18-493e-bf9b-7fbb328e01a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:45.078995 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.078971 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/550e0c64-4028-4fb3-98ef-fdc52cc66189-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:45.079056 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.079004 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2776ed01-5f18-493e-bf9b-7fbb328e01a2-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:45.079056 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.079017 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2776ed01-5f18-493e-bf9b-7fbb328e01a2-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:45.079056 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.079032 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/550e0c64-4028-4fb3-98ef-fdc52cc66189-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:45.079056 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.079045 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2776ed01-5f18-493e-bf9b-7fbb328e01a2-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:45.079198 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.079058 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2776ed01-5f18-493e-bf9b-7fbb328e01a2-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:45.079198 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.079074 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlprd\" (UniqueName: \"kubernetes.io/projected/550e0c64-4028-4fb3-98ef-fdc52cc66189-kube-api-access-nlprd\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:45.079198 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.079088 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2776ed01-5f18-493e-bf9b-7fbb328e01a2-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:45.079198 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.079101 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/550e0c64-4028-4fb3-98ef-fdc52cc66189-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:45.080050 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.080020 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2776ed01-5f18-493e-bf9b-7fbb328e01a2-istio-token" pod "2776ed01-5f18-493e-bf9b-7fbb328e01a2" (UID: "2776ed01-5f18-493e-bf9b-7fbb328e01a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:37:45.080050 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.080036 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2776ed01-5f18-493e-bf9b-7fbb328e01a2-istio-podinfo" pod "2776ed01-5f18-493e-bf9b-7fbb328e01a2" (UID: "2776ed01-5f18-493e-bf9b-7fbb328e01a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:37:45.080587 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.080564 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2776ed01-5f18-493e-bf9b-7fbb328e01a2-istio-envoy" pod "2776ed01-5f18-493e-bf9b-7fbb328e01a2" (UID: "2776ed01-5f18-493e-bf9b-7fbb328e01a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:45.080587 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.080578 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2776ed01-5f18-493e-bf9b-7fbb328e01a2-kube-api-access-cxfmv" pod "2776ed01-5f18-493e-bf9b-7fbb328e01a2" (UID: "2776ed01-5f18-493e-bf9b-7fbb328e01a2"). InnerVolumeSpecName "kube-api-access-cxfmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:37:45.180475 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.180439 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2776ed01-5f18-493e-bf9b-7fbb328e01a2-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:45.180475 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.180466 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxfmv\" (UniqueName: \"kubernetes.io/projected/2776ed01-5f18-493e-bf9b-7fbb328e01a2-kube-api-access-cxfmv\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:45.180475 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.180476 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2776ed01-5f18-493e-bf9b-7fbb328e01a2-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:45.180698 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.180485 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2776ed01-5f18-493e-bf9b-7fbb328e01a2-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:45.464962 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.464936 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" Oct 30 13:37:45.464962 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.464945 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4" event={"ID":"2776ed01-5f18-493e-bf9b-7fbb328e01a2","Type":"ContainerDied","Data":"502deb6ac88a3570a10e2f405450bbf3c366901d8495e90e8c7945094faf30cb"} Oct 30 13:37:45.465560 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.465002 4931 scope.go:122] "RemoveContainer" containerID="abe9d4324eceb24becf64e846066d1bd10981698ac0970ab00f5089dd831a6e5" Oct 30 13:37:45.466745 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.466722 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" Oct 30 13:37:45.466844 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.466762 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79" event={"ID":"8bc1a121-7dc1-4ccd-a395-68929a6f6ad1","Type":"ContainerDied","Data":"82d5b8e5f9169727f72a1964ca2e3e6cc086612d66405896a902e37de6beb5bd"} Oct 30 13:37:45.468913 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.468882 4931 generic.go:356] "Generic (PLEG): container finished" podID="550e0c64-4028-4fb3-98ef-fdc52cc66189" containerID="8c3e9c17377496135ef0edd2b29975097898c4bb897436fdd62c038eab997056" exitCode=0 Oct 30 13:37:45.469055 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.469015 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" event={"ID":"550e0c64-4028-4fb3-98ef-fdc52cc66189","Type":"ContainerDied","Data":"8c3e9c17377496135ef0edd2b29975097898c4bb897436fdd62c038eab997056"} Oct 30 13:37:45.469118 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.469093 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" event={"ID":"550e0c64-4028-4fb3-98ef-fdc52cc66189","Type":"ContainerDied","Data":"d202387ecdf26573391309111cf3d5799c08c2a5e0ddf01732ee11ef67096b5c"} Oct 30 13:37:45.471300 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.469216 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7" Oct 30 13:37:45.490089 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.486239 4931 scope.go:122] "RemoveContainer" containerID="233b07ef31af5a34f76e1a82d0b290c0ff3ca24a2a1093c0b05c79e175c5c1bd" Oct 30 13:37:45.498296 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.498238 4931 scope.go:122] "RemoveContainer" containerID="8c3e9c17377496135ef0edd2b29975097898c4bb897436fdd62c038eab997056" Oct 30 13:37:45.508611 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.508593 4931 scope.go:122] "RemoveContainer" containerID="8c3e9c17377496135ef0edd2b29975097898c4bb897436fdd62c038eab997056" Oct 30 13:37:45.508881 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:37:45.508858 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c3e9c17377496135ef0edd2b29975097898c4bb897436fdd62c038eab997056\": container with ID starting with 8c3e9c17377496135ef0edd2b29975097898c4bb897436fdd62c038eab997056 not found: ID does not exist" containerID="8c3e9c17377496135ef0edd2b29975097898c4bb897436fdd62c038eab997056" Oct 30 13:37:45.508980 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.508886 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c3e9c17377496135ef0edd2b29975097898c4bb897436fdd62c038eab997056"} err="failed to get container status \"8c3e9c17377496135ef0edd2b29975097898c4bb897436fdd62c038eab997056\": rpc error: code = NotFound desc = could not find container \"8c3e9c17377496135ef0edd2b29975097898c4bb897436fdd62c038eab997056\": container with ID starting with 8c3e9c17377496135ef0edd2b29975097898c4bb897436fdd62c038eab997056 not found: ID does not exist" Oct 30 13:37:45.537109 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.537081 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7"] Oct 30 13:37:45.549802 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.549773 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7"] Oct 30 13:37:45.580631 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.580596 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4"] Oct 30 13:37:45.584776 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.584753 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4"] Oct 30 13:37:45.620957 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.620923 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79"] Oct 30 13:37:45.627718 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:45.627685 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79"] Oct 30 13:37:46.149379 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:46.149301 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-srqbd"] Oct 30 13:37:46.149988 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:46.149973 4931 cpu_manager.go:414] "containerMap: removing container" podUID="2776ed01-5f18-493e-bf9b-7fbb328e01a2" containerName="istio-proxy" Oct 30 13:37:46.150031 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:46.149991 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2776ed01-5f18-493e-bf9b-7fbb328e01a2" containerName="istio-proxy" Oct 30 13:37:46.150031 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:46.150010 4931 cpu_manager.go:414] "containerMap: removing container" podUID="8bc1a121-7dc1-4ccd-a395-68929a6f6ad1" containerName="istio-proxy" Oct 30 13:37:46.150031 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:46.150015 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bc1a121-7dc1-4ccd-a395-68929a6f6ad1" containerName="istio-proxy" Oct 30 13:37:46.150031 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:46.150030 4931 cpu_manager.go:414] "containerMap: removing container" podUID="550e0c64-4028-4fb3-98ef-fdc52cc66189" containerName="istio-proxy" Oct 30 13:37:46.150167 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:46.150036 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="550e0c64-4028-4fb3-98ef-fdc52cc66189" containerName="istio-proxy" Oct 30 13:37:46.150167 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:46.150130 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="550e0c64-4028-4fb3-98ef-fdc52cc66189" containerName="istio-proxy" Oct 30 13:37:46.150167 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:46.150146 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="2776ed01-5f18-493e-bf9b-7fbb328e01a2" containerName="istio-proxy" Oct 30 13:37:46.150167 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:46.150154 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bc1a121-7dc1-4ccd-a395-68929a6f6ad1" containerName="istio-proxy" Oct 30 13:37:46.155541 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:46.155518 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-srqbd" Oct 30 13:37:46.172949 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:46.172915 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:37:46.174476 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:46.174454 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:37:46.176231 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:46.176206 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-srqbd"] Oct 30 13:37:46.308612 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:46.308575 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6141898a-d732-491e-aca2-ad76f1acb683-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3b47e7-srqbd\" (UID: \"6141898a-d732-491e-aca2-ad76f1acb683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-srqbd" Oct 30 13:37:46.308777 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:46.308641 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xx2h\" (UniqueName: \"kubernetes.io/projected/6141898a-d732-491e-aca2-ad76f1acb683-kube-api-access-8xx2h\") pod \"maas-api-revoke-keys-e2e-ipp-a-3b47e7-srqbd\" (UID: \"6141898a-d732-491e-aca2-ad76f1acb683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-srqbd" Oct 30 13:37:46.308777 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:46.308763 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6141898a-d732-491e-aca2-ad76f1acb683-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3b47e7-srqbd\" (UID: \"6141898a-d732-491e-aca2-ad76f1acb683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-srqbd" Oct 30 13:37:46.410336 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:46.410214 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6141898a-d732-491e-aca2-ad76f1acb683-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3b47e7-srqbd\" (UID: \"6141898a-d732-491e-aca2-ad76f1acb683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-srqbd" Oct 30 13:37:46.410336 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:46.410319 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xx2h\" (UniqueName: \"kubernetes.io/projected/6141898a-d732-491e-aca2-ad76f1acb683-kube-api-access-8xx2h\") pod \"maas-api-revoke-keys-e2e-ipp-a-3b47e7-srqbd\" (UID: \"6141898a-d732-491e-aca2-ad76f1acb683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-srqbd" Oct 30 13:37:46.410568 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:46.410379 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6141898a-d732-491e-aca2-ad76f1acb683-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3b47e7-srqbd\" (UID: \"6141898a-d732-491e-aca2-ad76f1acb683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-srqbd" Oct 30 13:37:46.411055 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:46.411033 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6141898a-d732-491e-aca2-ad76f1acb683-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3b47e7-srqbd\" (UID: \"6141898a-d732-491e-aca2-ad76f1acb683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-srqbd" Oct 30 13:37:46.412685 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:46.412658 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6141898a-d732-491e-aca2-ad76f1acb683-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3b47e7-srqbd\" (UID: \"6141898a-d732-491e-aca2-ad76f1acb683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-srqbd" Oct 30 13:37:46.435835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:46.435809 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xx2h\" (UniqueName: \"kubernetes.io/projected/6141898a-d732-491e-aca2-ad76f1acb683-kube-api-access-8xx2h\") pod \"maas-api-revoke-keys-e2e-ipp-a-3b47e7-srqbd\" (UID: \"6141898a-d732-491e-aca2-ad76f1acb683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-srqbd" Oct 30 13:37:46.470347 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:46.470310 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-srqbd" Oct 30 13:37:46.801027 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:37:46.800996 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6141898a_d732_491e_aca2_ad76f1acb683.slice/crio-9d88c08a97254c4745168b92e9814a18c2cdc80af16d4f446a43eb5e324aacbb WatchSource:0}: Error finding container 9d88c08a97254c4745168b92e9814a18c2cdc80af16d4f446a43eb5e324aacbb: Status 404 returned error can't find the container with id 9d88c08a97254c4745168b92e9814a18c2cdc80af16d4f446a43eb5e324aacbb Oct 30 13:37:46.802346 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:46.802322 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-srqbd"] Oct 30 13:37:47.490977 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:47.490938 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-srqbd" event={"ID":"6141898a-d732-491e-aca2-ad76f1acb683","Type":"ContainerStarted","Data":"6bdcc7a3fc7ce304c22f418b0d76cef2f467f163526c873d3e5b19f2a5639a82"} Oct 30 13:37:47.491419 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:47.490996 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-srqbd" event={"ID":"6141898a-d732-491e-aca2-ad76f1acb683","Type":"ContainerStarted","Data":"9d88c08a97254c4745168b92e9814a18c2cdc80af16d4f446a43eb5e324aacbb"} Oct 30 13:37:47.515031 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:47.514998 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2776ed01-5f18-493e-bf9b-7fbb328e01a2" path="/var/lib/kubelet/pods/2776ed01-5f18-493e-bf9b-7fbb328e01a2/volumes" Oct 30 13:37:47.515673 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:47.515652 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="550e0c64-4028-4fb3-98ef-fdc52cc66189" path="/var/lib/kubelet/pods/550e0c64-4028-4fb3-98ef-fdc52cc66189/volumes" Oct 30 13:37:47.516321 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:47.516305 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bc1a121-7dc1-4ccd-a395-68929a6f6ad1" path="/var/lib/kubelet/pods/8bc1a121-7dc1-4ccd-a395-68929a6f6ad1/volumes" Oct 30 13:37:47.542516 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:47.542470 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-srqbd" podStartSLOduration=1.542458736 podStartE2EDuration="1.542458736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:47.536840313 +0000 UTC m=+3166.140338759" watchObservedRunningTime="2026-10-30 13:37:47.542458736 +0000 UTC m=+3166.145957232" Oct 30 13:37:48.500402 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:48.500366 4931 generic.go:356] "Generic (PLEG): container finished" podID="6141898a-d732-491e-aca2-ad76f1acb683" containerID="6bdcc7a3fc7ce304c22f418b0d76cef2f467f163526c873d3e5b19f2a5639a82" exitCode=0 Oct 30 13:37:48.500821 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:48.500446 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-srqbd" event={"ID":"6141898a-d732-491e-aca2-ad76f1acb683","Type":"ContainerDied","Data":"6bdcc7a3fc7ce304c22f418b0d76cef2f467f163526c873d3e5b19f2a5639a82"} Oct 30 13:37:49.677680 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:49.677652 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-srqbd" Oct 30 13:37:49.774461 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:49.774365 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6141898a-d732-491e-aca2-ad76f1acb683-kube-api-access-8xx2h\" (UniqueName: \"kubernetes.io/projected/6141898a-d732-491e-aca2-ad76f1acb683-kube-api-access-8xx2h\") pod \"6141898a-d732-491e-aca2-ad76f1acb683\" (UID: \"6141898a-d732-491e-aca2-ad76f1acb683\") " Oct 30 13:37:49.774624 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:49.774501 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6141898a-d732-491e-aca2-ad76f1acb683-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6141898a-d732-491e-aca2-ad76f1acb683-maas-api-xks-ca\") pod \"6141898a-d732-491e-aca2-ad76f1acb683\" (UID: \"6141898a-d732-491e-aca2-ad76f1acb683\") " Oct 30 13:37:49.774685 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:49.774640 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6141898a-d732-491e-aca2-ad76f1acb683-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6141898a-d732-491e-aca2-ad76f1acb683-maas-api-openshift-ca\") pod \"6141898a-d732-491e-aca2-ad76f1acb683\" (UID: \"6141898a-d732-491e-aca2-ad76f1acb683\") " Oct 30 13:37:49.775101 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:49.775065 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6141898a-d732-491e-aca2-ad76f1acb683-maas-api-openshift-ca" pod "6141898a-d732-491e-aca2-ad76f1acb683" (UID: "6141898a-d732-491e-aca2-ad76f1acb683"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:37:49.775579 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:49.775540 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6141898a-d732-491e-aca2-ad76f1acb683-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:49.777014 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:49.776973 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6141898a-d732-491e-aca2-ad76f1acb683-kube-api-access-8xx2h" pod "6141898a-d732-491e-aca2-ad76f1acb683" (UID: "6141898a-d732-491e-aca2-ad76f1acb683"). InnerVolumeSpecName "kube-api-access-8xx2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:37:49.777415 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:49.777386 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6141898a-d732-491e-aca2-ad76f1acb683-maas-api-xks-ca" pod "6141898a-d732-491e-aca2-ad76f1acb683" (UID: "6141898a-d732-491e-aca2-ad76f1acb683"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:37:49.877374 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:49.877340 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xx2h\" (UniqueName: \"kubernetes.io/projected/6141898a-d732-491e-aca2-ad76f1acb683-kube-api-access-8xx2h\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:49.877374 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:49.877369 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6141898a-d732-491e-aca2-ad76f1acb683-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:50.572286 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:50.567620 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-srqbd" event={"ID":"6141898a-d732-491e-aca2-ad76f1acb683","Type":"ContainerDied","Data":"9d88c08a97254c4745168b92e9814a18c2cdc80af16d4f446a43eb5e324aacbb"} Oct 30 13:37:50.572286 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:50.567691 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d88c08a97254c4745168b92e9814a18c2cdc80af16d4f446a43eb5e324aacbb" Oct 30 13:37:50.572286 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:50.567760 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-srqbd" Oct 30 13:37:51.035603 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:51.035540 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-558a020d-x9w4g"] Oct 30 13:37:51.048934 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:51.048906 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-558a020d-x9w4g"] Oct 30 13:37:51.307949 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:51.307861 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:37:51.516426 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:51.516394 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22a99a6c-3118-44f4-b80a-cab1053c0b36" path="/var/lib/kubelet/pods/22a99a6c-3118-44f4-b80a-cab1053c0b36/volumes" Oct 30 13:37:55.041575 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:55.041532 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7bd69dd-m7lbc"] Oct 30 13:37:55.052290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:55.052251 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7bd69dd-m7lbc"] Oct 30 13:37:55.514765 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:55.514731 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="676089c8-e20a-4aa2-8220-be872d92bead" path="/var/lib/kubelet/pods/676089c8-e20a-4aa2-8220-be872d92bead/volumes" Oct 30 13:37:59.403798 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:59.403737 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" podUID="65799c14-8de4-4b4a-90df-7cd257e1e581" containerName="main" containerID="cri-o://d97fc7e4e5cbc5ef1300d36aa84ea9e353aaec9f6d23bbeb29a96baab879d54e" gracePeriod=45 Oct 30 13:37:59.538233 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:59.538198 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" podUID="65799c14-8de4-4b4a-90df-7cd257e1e581" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": dial tcp 10.132.0.63:8000: connect: connection refused" Oct 30 13:37:59.641327 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:59.641259 4931 generic.go:356] "Generic (PLEG): container finished" podID="65799c14-8de4-4b4a-90df-7cd257e1e581" containerID="d97fc7e4e5cbc5ef1300d36aa84ea9e353aaec9f6d23bbeb29a96baab879d54e" exitCode=0 Oct 30 13:37:59.641327 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:59.641321 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" event={"ID":"65799c14-8de4-4b4a-90df-7cd257e1e581","Type":"ContainerDied","Data":"d97fc7e4e5cbc5ef1300d36aa84ea9e353aaec9f6d23bbeb29a96baab879d54e"} Oct 30 13:37:59.696353 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:59.696327 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:37:59.727884 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:59.727839 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65799c14-8de4-4b4a-90df-7cd257e1e581-dshm\" (UniqueName: \"kubernetes.io/empty-dir/65799c14-8de4-4b4a-90df-7cd257e1e581-dshm\") pod \"65799c14-8de4-4b4a-90df-7cd257e1e581\" (UID: \"65799c14-8de4-4b4a-90df-7cd257e1e581\") " Oct 30 13:37:59.728075 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:59.727953 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65799c14-8de4-4b4a-90df-7cd257e1e581-tls-certs\" (UniqueName: \"kubernetes.io/secret/65799c14-8de4-4b4a-90df-7cd257e1e581-tls-certs\") pod \"65799c14-8de4-4b4a-90df-7cd257e1e581\" (UID: \"65799c14-8de4-4b4a-90df-7cd257e1e581\") " Oct 30 13:37:59.728075 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:59.727991 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65799c14-8de4-4b4a-90df-7cd257e1e581-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65799c14-8de4-4b4a-90df-7cd257e1e581-tmp-dir\") pod \"65799c14-8de4-4b4a-90df-7cd257e1e581\" (UID: \"65799c14-8de4-4b4a-90df-7cd257e1e581\") " Oct 30 13:37:59.728075 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:59.728012 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65799c14-8de4-4b4a-90df-7cd257e1e581-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65799c14-8de4-4b4a-90df-7cd257e1e581-model-cache\") pod \"65799c14-8de4-4b4a-90df-7cd257e1e581\" (UID: \"65799c14-8de4-4b4a-90df-7cd257e1e581\") " Oct 30 13:37:59.728309 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:59.728133 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65799c14-8de4-4b4a-90df-7cd257e1e581-kube-api-access-47zqt\" (UniqueName: \"kubernetes.io/projected/65799c14-8de4-4b4a-90df-7cd257e1e581-kube-api-access-47zqt\") pod \"65799c14-8de4-4b4a-90df-7cd257e1e581\" (UID: \"65799c14-8de4-4b4a-90df-7cd257e1e581\") " Oct 30 13:37:59.728309 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:59.728165 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65799c14-8de4-4b4a-90df-7cd257e1e581-home\" (UniqueName: \"kubernetes.io/empty-dir/65799c14-8de4-4b4a-90df-7cd257e1e581-home\") pod \"65799c14-8de4-4b4a-90df-7cd257e1e581\" (UID: \"65799c14-8de4-4b4a-90df-7cd257e1e581\") " Oct 30 13:37:59.728309 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:59.728245 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65799c14-8de4-4b4a-90df-7cd257e1e581-tmp-dir" pod "65799c14-8de4-4b4a-90df-7cd257e1e581" (UID: "65799c14-8de4-4b4a-90df-7cd257e1e581"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:59.728486 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:59.728344 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65799c14-8de4-4b4a-90df-7cd257e1e581-model-cache" pod "65799c14-8de4-4b4a-90df-7cd257e1e581" (UID: "65799c14-8de4-4b4a-90df-7cd257e1e581"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:59.728540 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:59.728508 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65799c14-8de4-4b4a-90df-7cd257e1e581-home" pod "65799c14-8de4-4b4a-90df-7cd257e1e581" (UID: "65799c14-8de4-4b4a-90df-7cd257e1e581"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:59.728872 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:59.728850 4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65799c14-8de4-4b4a-90df-7cd257e1e581-tmp-dir\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:59.728872 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:59.728869 4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65799c14-8de4-4b4a-90df-7cd257e1e581-model-cache\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:59.729042 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:59.728878 4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65799c14-8de4-4b4a-90df-7cd257e1e581-home\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:59.730360 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:59.730333 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65799c14-8de4-4b4a-90df-7cd257e1e581-dshm" pod "65799c14-8de4-4b4a-90df-7cd257e1e581" (UID: "65799c14-8de4-4b4a-90df-7cd257e1e581"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:59.730481 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:59.730397 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65799c14-8de4-4b4a-90df-7cd257e1e581-kube-api-access-47zqt" pod "65799c14-8de4-4b4a-90df-7cd257e1e581" (UID: "65799c14-8de4-4b4a-90df-7cd257e1e581"). InnerVolumeSpecName "kube-api-access-47zqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:37:59.730481 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:59.730411 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65799c14-8de4-4b4a-90df-7cd257e1e581-tls-certs" pod "65799c14-8de4-4b4a-90df-7cd257e1e581" (UID: "65799c14-8de4-4b4a-90df-7cd257e1e581"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:37:59.831212 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:59.831182 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47zqt\" (UniqueName: \"kubernetes.io/projected/65799c14-8de4-4b4a-90df-7cd257e1e581-kube-api-access-47zqt\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:59.831212 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:59.831211 4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65799c14-8de4-4b4a-90df-7cd257e1e581-dshm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:37:59.831448 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:37:59.831223 4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65799c14-8de4-4b4a-90df-7cd257e1e581-tls-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:00.651232 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:00.651197 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" event={"ID":"65799c14-8de4-4b4a-90df-7cd257e1e581","Type":"ContainerDied","Data":"580a6913d9ddc92ff6ec1dc0e8d9c8da8006c1acfbb44c4304a06c64c2ab88f8"} Oct 30 13:38:00.651232 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:00.651229 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2" Oct 30 13:38:00.651774 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:00.651253 4931 scope.go:122] "RemoveContainer" containerID="d97fc7e4e5cbc5ef1300d36aa84ea9e353aaec9f6d23bbeb29a96baab879d54e" Oct 30 13:38:00.711662 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:00.711618 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2"] Oct 30 13:38:00.724457 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:00.724426 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-09db16/premium-w3-09db16-kserve-59f68cb797-zjvc2"] Oct 30 13:38:01.036601 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:01.036566 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1981ad18-r7phw"] Oct 30 13:38:01.041040 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:01.041011 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1981ad18-r7phw"] Oct 30 13:38:01.521733 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:01.521690 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65799c14-8de4-4b4a-90df-7cd257e1e581" path="/var/lib/kubelet/pods/65799c14-8de4-4b4a-90df-7cd257e1e581/volumes" Oct 30 13:38:01.522551 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:01.522522 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83ca9233-afe0-4d73-b7dc-f969b928fdcb" path="/var/lib/kubelet/pods/83ca9233-afe0-4d73-b7dc-f969b928fdcb/volumes" Oct 30 13:38:06.033634 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:06.033600 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fc55545-bwn9x"] Oct 30 13:38:06.046457 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:06.046425 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fc55545-bwn9x"] Oct 30 13:38:07.354094 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:38:07.354058 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:38:07.514612 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:07.514580 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="186079bb-e477-4fee-b293-54356a7e6310" path="/var/lib/kubelet/pods/186079bb-e477-4fee-b293-54356a7e6310/volumes" Oct 30 13:38:18.741829 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:18.741735 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp"] Oct 30 13:38:18.742334 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:18.742001 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp" podUID="44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b" containerName="maas-api" containerID="cri-o://1bf3296060610f9f5a422dc7215731e3ffdf6ffb1dd84570e15015713a359bf1" gracePeriod=30 Oct 30 13:38:18.895889 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:18.895800 4931 generic.go:356] "Generic (PLEG): container finished" podID="44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b" containerID="1bf3296060610f9f5a422dc7215731e3ffdf6ffb1dd84570e15015713a359bf1" exitCode=0 Oct 30 13:38:18.896296 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:18.895865 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp" event={"ID":"44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b","Type":"ContainerDied","Data":"1bf3296060610f9f5a422dc7215731e3ffdf6ffb1dd84570e15015713a359bf1"} Oct 30 13:38:19.109571 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:19.109539 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp" Oct 30 13:38:19.129052 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:19.129016 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b-kube-api-access-xhktc\" (UniqueName: \"kubernetes.io/projected/44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b-kube-api-access-xhktc\") pod \"44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b\" (UID: \"44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b\") " Oct 30 13:38:19.129384 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:19.129355 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b-maas-api-tls\") pod \"44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b\" (UID: \"44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b\") " Oct 30 13:38:19.129690 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:19.129664 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b-openshift-service-ca\") pod \"44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b\" (UID: \"44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b\") " Oct 30 13:38:19.129872 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:19.129853 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b-metrics-certs\") pod \"44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b\" (UID: \"44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b\") " Oct 30 13:38:19.130040 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:19.130012 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b-openshift-service-ca" pod "44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b" (UID: "44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:38:19.130956 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:19.130932 4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b-openshift-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:19.132467 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:19.131946 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b-kube-api-access-xhktc" pod "44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b" (UID: "44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b"). InnerVolumeSpecName "kube-api-access-xhktc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:38:19.132467 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:19.132401 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b-maas-api-tls" pod "44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b" (UID: "44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:38:19.132829 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:19.132807 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b-metrics-certs" pod "44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b" (UID: "44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:38:19.232717 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:19.232683 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhktc\" (UniqueName: \"kubernetes.io/projected/44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b-kube-api-access-xhktc\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:19.232717 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:19.232715 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b-maas-api-tls\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:19.232717 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:19.232725 4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b-metrics-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:19.906603 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:19.906523 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp" Oct 30 13:38:19.906603 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:19.906538 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp" event={"ID":"44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b","Type":"ContainerDied","Data":"61f3e420138ecec977ccf36cbc0d50fda0cc775a0855923a697e83b1e9816ee9"} Oct 30 13:38:19.907073 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:19.906639 4931 scope.go:122] "RemoveContainer" containerID="1bf3296060610f9f5a422dc7215731e3ffdf6ffb1dd84570e15015713a359bf1" Oct 30 13:38:19.971175 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:19.971144 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp"] Oct 30 13:38:19.977229 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:19.977202 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f385-5d85649b6d-qqtpp"] Oct 30 13:38:21.550372 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:21.550335 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b" path="/var/lib/kubelet/pods/44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b/volumes" Oct 30 13:38:22.191349 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:22.191310 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng"] Oct 30 13:38:22.193667 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:22.192470 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng" podUID="20802417-843e-455e-a55f-b0972b386ac6" containerName="payload-pre-processing" containerID="cri-o://380e66aa97d55fe278e930828429ff5ceb2a371fe5f0b27130756c3de1059cb9" gracePeriod=30 Oct 30 13:38:22.800492 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:22.799713 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng" Oct 30 13:38:22.875303 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:22.870988 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20802417-843e-455e-a55f-b0972b386ac6-kube-api-access-7bf6m\" (UniqueName: \"kubernetes.io/projected/20802417-843e-455e-a55f-b0972b386ac6-kube-api-access-7bf6m\") pod \"20802417-843e-455e-a55f-b0972b386ac6\" (UID: \"20802417-843e-455e-a55f-b0972b386ac6\") " Oct 30 13:38:22.875303 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:22.871120 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20802417-843e-455e-a55f-b0972b386ac6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20802417-843e-455e-a55f-b0972b386ac6-plugins-config-volume\") pod \"20802417-843e-455e-a55f-b0972b386ac6\" (UID: \"20802417-843e-455e-a55f-b0972b386ac6\") " Oct 30 13:38:22.875303 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:22.871851 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20802417-843e-455e-a55f-b0972b386ac6-plugins-config-volume" pod "20802417-843e-455e-a55f-b0972b386ac6" (UID: "20802417-843e-455e-a55f-b0972b386ac6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:38:22.875303 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:22.872441 4931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20802417-843e-455e-a55f-b0972b386ac6-plugins-config-volume\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:22.896663 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:22.896600 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20802417-843e-455e-a55f-b0972b386ac6-kube-api-access-7bf6m" pod "20802417-843e-455e-a55f-b0972b386ac6" (UID: "20802417-843e-455e-a55f-b0972b386ac6"). InnerVolumeSpecName "kube-api-access-7bf6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:38:22.974004 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:22.973968 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bf6m\" (UniqueName: \"kubernetes.io/projected/20802417-843e-455e-a55f-b0972b386ac6-kube-api-access-7bf6m\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:22.999359 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:22.999323 4931 generic.go:356] "Generic (PLEG): container finished" podID="20802417-843e-455e-a55f-b0972b386ac6" containerID="380e66aa97d55fe278e930828429ff5ceb2a371fe5f0b27130756c3de1059cb9" exitCode=0 Oct 30 13:38:22.999533 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:22.999365 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng" Oct 30 13:38:22.999533 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:22.999422 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng" event={"ID":"20802417-843e-455e-a55f-b0972b386ac6","Type":"ContainerDied","Data":"380e66aa97d55fe278e930828429ff5ceb2a371fe5f0b27130756c3de1059cb9"} Oct 30 13:38:22.999533 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:22.999480 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng" event={"ID":"20802417-843e-455e-a55f-b0972b386ac6","Type":"ContainerDied","Data":"3307156ccb940260ac91408b679f7023399cfc29b84693ae640b339a2ad21c92"} Oct 30 13:38:22.999533 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:22.999503 4931 scope.go:122] "RemoveContainer" containerID="380e66aa97d55fe278e930828429ff5ceb2a371fe5f0b27130756c3de1059cb9" Oct 30 13:38:23.010688 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:23.010671 4931 scope.go:122] "RemoveContainer" containerID="380e66aa97d55fe278e930828429ff5ceb2a371fe5f0b27130756c3de1059cb9" Oct 30 13:38:23.010957 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:38:23.010934 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"380e66aa97d55fe278e930828429ff5ceb2a371fe5f0b27130756c3de1059cb9\": container with ID starting with 380e66aa97d55fe278e930828429ff5ceb2a371fe5f0b27130756c3de1059cb9 not found: ID does not exist" containerID="380e66aa97d55fe278e930828429ff5ceb2a371fe5f0b27130756c3de1059cb9" Oct 30 13:38:23.011050 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:23.010967 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"380e66aa97d55fe278e930828429ff5ceb2a371fe5f0b27130756c3de1059cb9"} err="failed to get container status \"380e66aa97d55fe278e930828429ff5ceb2a371fe5f0b27130756c3de1059cb9\": rpc error: code = NotFound desc = could not find container \"380e66aa97d55fe278e930828429ff5ceb2a371fe5f0b27130756c3de1059cb9\": container with ID starting with 380e66aa97d55fe278e930828429ff5ceb2a371fe5f0b27130756c3de1059cb9 not found: ID does not exist" Oct 30 13:38:23.096411 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:23.096335 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng"] Oct 30 13:38:23.104065 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:23.104002 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng"] Oct 30 13:38:23.514808 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:23.514733 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20802417-843e-455e-a55f-b0972b386ac6" path="/var/lib/kubelet/pods/20802417-843e-455e-a55f-b0972b386ac6/volumes" Oct 30 13:38:24.137312 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:24.137259 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-7cxbc"] Oct 30 13:38:24.138019 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:24.138002 4931 cpu_manager.go:414] "containerMap: removing container" podUID="6141898a-d732-491e-aca2-ad76f1acb683" containerName="revoke-keys" Oct 30 13:38:24.138084 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:24.138020 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6141898a-d732-491e-aca2-ad76f1acb683" containerName="revoke-keys" Oct 30 13:38:24.138084 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:24.138038 4931 cpu_manager.go:414] "containerMap: removing container" podUID="20802417-843e-455e-a55f-b0972b386ac6" containerName="payload-pre-processing" Oct 30 13:38:24.138084 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:24.138045 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20802417-843e-455e-a55f-b0972b386ac6" containerName="payload-pre-processing" Oct 30 13:38:24.138084 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:24.138076 4931 cpu_manager.go:414] "containerMap: removing container" podUID="44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b" containerName="maas-api" Oct 30 13:38:24.138084 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:24.138082 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b" containerName="maas-api" Oct 30 13:38:24.138255 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:24.138090 4931 cpu_manager.go:414] "containerMap: removing container" podUID="65799c14-8de4-4b4a-90df-7cd257e1e581" containerName="main" Oct 30 13:38:24.138255 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:24.138095 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65799c14-8de4-4b4a-90df-7cd257e1e581" containerName="main" Oct 30 13:38:24.138255 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:24.138212 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6141898a-d732-491e-aca2-ad76f1acb683" containerName="revoke-keys" Oct 30 13:38:24.138255 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:24.138221 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="20802417-843e-455e-a55f-b0972b386ac6" containerName="payload-pre-processing" Oct 30 13:38:24.138255 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:24.138233 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="44b6c7ea-cef8-448d-b6ee-5b8ca9063b3b" containerName="maas-api" Oct 30 13:38:24.138255 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:24.138240 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="65799c14-8de4-4b4a-90df-7cd257e1e581" containerName="main" Oct 30 13:38:24.141907 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:24.141888 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-7cxbc" Oct 30 13:38:24.171670 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:24.170996 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:38:24.171895 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:24.171613 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:38:24.178936 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:24.178906 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-7cxbc"] Oct 30 13:38:24.205459 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:24.205420 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/607401ec-30ee-476c-bbb8-e3e90c9fec75-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-78543b-7cxbc\" (UID: \"607401ec-30ee-476c-bbb8-e3e90c9fec75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-7cxbc" Oct 30 13:38:24.205629 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:24.205483 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/607401ec-30ee-476c-bbb8-e3e90c9fec75-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-78543b-7cxbc\" (UID: \"607401ec-30ee-476c-bbb8-e3e90c9fec75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-7cxbc" Oct 30 13:38:24.205702 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:24.205634 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlzhn\" (UniqueName: \"kubernetes.io/projected/607401ec-30ee-476c-bbb8-e3e90c9fec75-kube-api-access-mlzhn\") pod \"maas-api-revoke-keys-e2e-ipp-a-78543b-7cxbc\" (UID: \"607401ec-30ee-476c-bbb8-e3e90c9fec75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-7cxbc" Oct 30 13:38:24.306773 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:24.306734 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/607401ec-30ee-476c-bbb8-e3e90c9fec75-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-78543b-7cxbc\" (UID: \"607401ec-30ee-476c-bbb8-e3e90c9fec75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-7cxbc" Oct 30 13:38:24.306937 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:24.306806 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/607401ec-30ee-476c-bbb8-e3e90c9fec75-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-78543b-7cxbc\" (UID: \"607401ec-30ee-476c-bbb8-e3e90c9fec75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-7cxbc" Oct 30 13:38:24.306937 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:24.306923 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlzhn\" (UniqueName: \"kubernetes.io/projected/607401ec-30ee-476c-bbb8-e3e90c9fec75-kube-api-access-mlzhn\") pod \"maas-api-revoke-keys-e2e-ipp-a-78543b-7cxbc\" (UID: \"607401ec-30ee-476c-bbb8-e3e90c9fec75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-7cxbc" Oct 30 13:38:24.307551 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:24.307525 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/607401ec-30ee-476c-bbb8-e3e90c9fec75-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-78543b-7cxbc\" (UID: \"607401ec-30ee-476c-bbb8-e3e90c9fec75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-7cxbc" Oct 30 13:38:24.309652 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:24.309629 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/607401ec-30ee-476c-bbb8-e3e90c9fec75-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-78543b-7cxbc\" (UID: \"607401ec-30ee-476c-bbb8-e3e90c9fec75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-7cxbc" Oct 30 13:38:24.331873 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:24.331847 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlzhn\" (UniqueName: \"kubernetes.io/projected/607401ec-30ee-476c-bbb8-e3e90c9fec75-kube-api-access-mlzhn\") pod \"maas-api-revoke-keys-e2e-ipp-a-78543b-7cxbc\" (UID: \"607401ec-30ee-476c-bbb8-e3e90c9fec75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-7cxbc" Oct 30 13:38:24.458243 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:24.458214 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-7cxbc" Oct 30 13:38:24.784087 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:24.784037 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-7cxbc"] Oct 30 13:38:25.017336 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:25.017296 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-7cxbc" event={"ID":"607401ec-30ee-476c-bbb8-e3e90c9fec75","Type":"ContainerStarted","Data":"ec41109d5c70197c293d4123166cab15e586942e07b27cd7fb17aefc5830c0db"} Oct 30 13:38:26.027112 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:26.027070 4931 generic.go:356] "Generic (PLEG): container finished" podID="607401ec-30ee-476c-bbb8-e3e90c9fec75" containerID="bf68d3e91f8b6f6e4659290ac9c5be396b2e754ee2a7f410ce091fc5ce952989" exitCode=0 Oct 30 13:38:26.027576 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:26.027155 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-7cxbc" event={"ID":"607401ec-30ee-476c-bbb8-e3e90c9fec75","Type":"ContainerDied","Data":"bf68d3e91f8b6f6e4659290ac9c5be396b2e754ee2a7f410ce091fc5ce952989"} Oct 30 13:38:27.223530 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:27.223500 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-7cxbc" Oct 30 13:38:27.249058 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:27.249026 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/607401ec-30ee-476c-bbb8-e3e90c9fec75-kube-api-access-mlzhn\" (UniqueName: \"kubernetes.io/projected/607401ec-30ee-476c-bbb8-e3e90c9fec75-kube-api-access-mlzhn\") pod \"607401ec-30ee-476c-bbb8-e3e90c9fec75\" (UID: \"607401ec-30ee-476c-bbb8-e3e90c9fec75\") " Oct 30 13:38:27.249231 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:27.249182 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/607401ec-30ee-476c-bbb8-e3e90c9fec75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/607401ec-30ee-476c-bbb8-e3e90c9fec75-maas-api-openshift-ca\") pod \"607401ec-30ee-476c-bbb8-e3e90c9fec75\" (UID: \"607401ec-30ee-476c-bbb8-e3e90c9fec75\") " Oct 30 13:38:27.249434 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:27.249363 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/607401ec-30ee-476c-bbb8-e3e90c9fec75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/607401ec-30ee-476c-bbb8-e3e90c9fec75-maas-api-xks-ca\") pod \"607401ec-30ee-476c-bbb8-e3e90c9fec75\" (UID: \"607401ec-30ee-476c-bbb8-e3e90c9fec75\") " Oct 30 13:38:27.249562 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:27.249543 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/607401ec-30ee-476c-bbb8-e3e90c9fec75-maas-api-openshift-ca" pod "607401ec-30ee-476c-bbb8-e3e90c9fec75" (UID: "607401ec-30ee-476c-bbb8-e3e90c9fec75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:38:27.249974 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:27.249952 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/607401ec-30ee-476c-bbb8-e3e90c9fec75-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:27.251215 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:27.251194 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/607401ec-30ee-476c-bbb8-e3e90c9fec75-kube-api-access-mlzhn" pod "607401ec-30ee-476c-bbb8-e3e90c9fec75" (UID: "607401ec-30ee-476c-bbb8-e3e90c9fec75"). InnerVolumeSpecName "kube-api-access-mlzhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:38:27.251396 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:27.251373 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/607401ec-30ee-476c-bbb8-e3e90c9fec75-maas-api-xks-ca" pod "607401ec-30ee-476c-bbb8-e3e90c9fec75" (UID: "607401ec-30ee-476c-bbb8-e3e90c9fec75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:38:27.351432 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:27.351344 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/607401ec-30ee-476c-bbb8-e3e90c9fec75-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:27.351662 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:27.351641 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlzhn\" (UniqueName: \"kubernetes.io/projected/607401ec-30ee-476c-bbb8-e3e90c9fec75-kube-api-access-mlzhn\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:27.911029 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:27.910953 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c91909-4rb57"] Oct 30 13:38:27.911658 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:27.911642 4931 cpu_manager.go:414] "containerMap: removing container" podUID="607401ec-30ee-476c-bbb8-e3e90c9fec75" containerName="revoke-keys" Oct 30 13:38:27.911715 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:27.911660 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="607401ec-30ee-476c-bbb8-e3e90c9fec75" containerName="revoke-keys" Oct 30 13:38:27.911772 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:27.911761 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="607401ec-30ee-476c-bbb8-e3e90c9fec75" containerName="revoke-keys" Oct 30 13:38:27.915512 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:27.915491 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c91909-4rb57" Oct 30 13:38:27.939725 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:27.939697 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c91909-4rb57"] Oct 30 13:38:27.974753 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:27.974716 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd73f227-55d8-4861-9845-1586fd021731-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c91909-4rb57\" (UID: \"fd73f227-55d8-4861-9845-1586fd021731\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c91909-4rb57" Oct 30 13:38:27.974971 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:27.974950 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crhv2\" (UniqueName: \"kubernetes.io/projected/fd73f227-55d8-4861-9845-1586fd021731-kube-api-access-crhv2\") pod \"maas-api-revoke-keys-e2e-ipp-b-c91909-4rb57\" (UID: \"fd73f227-55d8-4861-9845-1586fd021731\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c91909-4rb57" Oct 30 13:38:27.975090 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:27.975071 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd73f227-55d8-4861-9845-1586fd021731-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c91909-4rb57\" (UID: \"fd73f227-55d8-4861-9845-1586fd021731\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c91909-4rb57" Oct 30 13:38:28.054744 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:28.054711 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-7cxbc" Oct 30 13:38:28.054911 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:28.054754 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-7cxbc" event={"ID":"607401ec-30ee-476c-bbb8-e3e90c9fec75","Type":"ContainerDied","Data":"ec41109d5c70197c293d4123166cab15e586942e07b27cd7fb17aefc5830c0db"} Oct 30 13:38:28.054911 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:28.054844 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec41109d5c70197c293d4123166cab15e586942e07b27cd7fb17aefc5830c0db" Oct 30 13:38:28.076587 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:28.076557 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crhv2\" (UniqueName: \"kubernetes.io/projected/fd73f227-55d8-4861-9845-1586fd021731-kube-api-access-crhv2\") pod \"maas-api-revoke-keys-e2e-ipp-b-c91909-4rb57\" (UID: \"fd73f227-55d8-4861-9845-1586fd021731\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c91909-4rb57" Oct 30 13:38:28.076728 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:28.076645 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd73f227-55d8-4861-9845-1586fd021731-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c91909-4rb57\" (UID: \"fd73f227-55d8-4861-9845-1586fd021731\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c91909-4rb57" Oct 30 13:38:28.076772 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:28.076755 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd73f227-55d8-4861-9845-1586fd021731-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c91909-4rb57\" (UID: \"fd73f227-55d8-4861-9845-1586fd021731\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c91909-4rb57" Oct 30 13:38:28.077355 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:28.077336 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd73f227-55d8-4861-9845-1586fd021731-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c91909-4rb57\" (UID: \"fd73f227-55d8-4861-9845-1586fd021731\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c91909-4rb57" Oct 30 13:38:28.079032 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:28.079010 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd73f227-55d8-4861-9845-1586fd021731-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c91909-4rb57\" (UID: \"fd73f227-55d8-4861-9845-1586fd021731\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c91909-4rb57" Oct 30 13:38:28.104607 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:28.104586 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crhv2\" (UniqueName: \"kubernetes.io/projected/fd73f227-55d8-4861-9845-1586fd021731-kube-api-access-crhv2\") pod \"maas-api-revoke-keys-e2e-ipp-b-c91909-4rb57\" (UID: \"fd73f227-55d8-4861-9845-1586fd021731\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c91909-4rb57" Oct 30 13:38:28.229464 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:28.229435 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c91909-4rb57" Oct 30 13:38:28.545750 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:38:28.545715 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd73f227_55d8_4861_9845_1586fd021731.slice/crio-25a6d8c108cecf2702e01b2bb3ac652e3ddb366793b1858af84ba343eaf82a80 WatchSource:0}: Error finding container 25a6d8c108cecf2702e01b2bb3ac652e3ddb366793b1858af84ba343eaf82a80: Status 404 returned error can't find the container with id 25a6d8c108cecf2702e01b2bb3ac652e3ddb366793b1858af84ba343eaf82a80 Oct 30 13:38:28.545934 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:28.545876 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c91909-4rb57"] Oct 30 13:38:29.118076 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:29.117980 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c91909-4rb57" event={"ID":"fd73f227-55d8-4861-9845-1586fd021731","Type":"ContainerStarted","Data":"e187ce1424ddb8b4ff7f85f99dac8f235930df4945d56e3e888234425bec65d7"} Oct 30 13:38:29.118350 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:29.118323 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c91909-4rb57" event={"ID":"fd73f227-55d8-4861-9845-1586fd021731","Type":"ContainerStarted","Data":"25a6d8c108cecf2702e01b2bb3ac652e3ddb366793b1858af84ba343eaf82a80"} Oct 30 13:38:29.191067 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:29.191001 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c91909-4rb57" podStartSLOduration=2.190982549 podStartE2EDuration="2.190982549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:38:29.16559835 +0000 UTC m=+3207.769096797" watchObservedRunningTime="2026-10-30 13:38:29.190982549 +0000 UTC m=+3207.794480993" Oct 30 13:38:29.596110 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:29.596077 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-mq6wx"] Oct 30 13:38:29.600603 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:29.600577 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-mq6wx" Oct 30 13:38:29.623329 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:29.623298 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-mq6wx"] Oct 30 13:38:29.717138 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:29.717102 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deea9d6c-9b3a-40d5-9b90-74818449aed0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-72208f-mq6wx\" (UID: \"deea9d6c-9b3a-40d5-9b90-74818449aed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-mq6wx" Oct 30 13:38:29.717380 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:29.717245 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmqjm\" (UniqueName: \"kubernetes.io/projected/deea9d6c-9b3a-40d5-9b90-74818449aed0-kube-api-access-zmqjm\") pod \"maas-api-revoke-keys-e2e-ipp-b-72208f-mq6wx\" (UID: \"deea9d6c-9b3a-40d5-9b90-74818449aed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-mq6wx" Oct 30 13:38:29.717467 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:29.717442 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deea9d6c-9b3a-40d5-9b90-74818449aed0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-72208f-mq6wx\" (UID: \"deea9d6c-9b3a-40d5-9b90-74818449aed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-mq6wx" Oct 30 13:38:29.818512 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:29.818433 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deea9d6c-9b3a-40d5-9b90-74818449aed0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-72208f-mq6wx\" (UID: \"deea9d6c-9b3a-40d5-9b90-74818449aed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-mq6wx" Oct 30 13:38:29.818745 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:29.818553 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmqjm\" (UniqueName: \"kubernetes.io/projected/deea9d6c-9b3a-40d5-9b90-74818449aed0-kube-api-access-zmqjm\") pod \"maas-api-revoke-keys-e2e-ipp-b-72208f-mq6wx\" (UID: \"deea9d6c-9b3a-40d5-9b90-74818449aed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-mq6wx" Oct 30 13:38:29.818745 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:29.818674 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deea9d6c-9b3a-40d5-9b90-74818449aed0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-72208f-mq6wx\" (UID: \"deea9d6c-9b3a-40d5-9b90-74818449aed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-mq6wx" Oct 30 13:38:29.819284 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:29.819246 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deea9d6c-9b3a-40d5-9b90-74818449aed0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-72208f-mq6wx\" (UID: \"deea9d6c-9b3a-40d5-9b90-74818449aed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-mq6wx" Oct 30 13:38:29.821018 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:29.820989 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deea9d6c-9b3a-40d5-9b90-74818449aed0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-72208f-mq6wx\" (UID: \"deea9d6c-9b3a-40d5-9b90-74818449aed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-mq6wx" Oct 30 13:38:29.843299 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:29.843257 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmqjm\" (UniqueName: \"kubernetes.io/projected/deea9d6c-9b3a-40d5-9b90-74818449aed0-kube-api-access-zmqjm\") pod \"maas-api-revoke-keys-e2e-ipp-b-72208f-mq6wx\" (UID: \"deea9d6c-9b3a-40d5-9b90-74818449aed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-mq6wx" Oct 30 13:38:29.915873 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:29.915842 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-mq6wx" Oct 30 13:38:30.291438 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:30.290488 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-mq6wx"] Oct 30 13:38:30.291438 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:30.291036 4931 generic.go:356] "Generic (PLEG): container finished" podID="fd73f227-55d8-4861-9845-1586fd021731" containerID="e187ce1424ddb8b4ff7f85f99dac8f235930df4945d56e3e888234425bec65d7" exitCode=0 Oct 30 13:38:30.291438 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:30.291289 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c91909-4rb57" event={"ID":"fd73f227-55d8-4861-9845-1586fd021731","Type":"ContainerDied","Data":"e187ce1424ddb8b4ff7f85f99dac8f235930df4945d56e3e888234425bec65d7"} Oct 30 13:38:31.301488 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:31.301457 4931 generic.go:356] "Generic (PLEG): container finished" podID="deea9d6c-9b3a-40d5-9b90-74818449aed0" containerID="f2ad03511ca29a990448840f20a566bea86d7089846c4e48d1508719652fc588" exitCode=0 Oct 30 13:38:31.301894 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:31.301537 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-mq6wx" event={"ID":"deea9d6c-9b3a-40d5-9b90-74818449aed0","Type":"ContainerDied","Data":"f2ad03511ca29a990448840f20a566bea86d7089846c4e48d1508719652fc588"} Oct 30 13:38:31.301894 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:31.301588 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-mq6wx" event={"ID":"deea9d6c-9b3a-40d5-9b90-74818449aed0","Type":"ContainerStarted","Data":"528cec0087b0443cad1b54b74d5b4c923260f858298d9715be6790c4fac469a0"} Oct 30 13:38:31.574206 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:31.574166 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c91909-4rb57" Oct 30 13:38:31.701507 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:31.701470 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd73f227-55d8-4861-9845-1586fd021731-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd73f227-55d8-4861-9845-1586fd021731-maas-api-xks-ca\") pod \"fd73f227-55d8-4861-9845-1586fd021731\" (UID: \"fd73f227-55d8-4861-9845-1586fd021731\") " Oct 30 13:38:31.701706 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:31.701594 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd73f227-55d8-4861-9845-1586fd021731-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd73f227-55d8-4861-9845-1586fd021731-maas-api-openshift-ca\") pod \"fd73f227-55d8-4861-9845-1586fd021731\" (UID: \"fd73f227-55d8-4861-9845-1586fd021731\") " Oct 30 13:38:31.701706 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:31.701646 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd73f227-55d8-4861-9845-1586fd021731-kube-api-access-crhv2\" (UniqueName: \"kubernetes.io/projected/fd73f227-55d8-4861-9845-1586fd021731-kube-api-access-crhv2\") pod \"fd73f227-55d8-4861-9845-1586fd021731\" (UID: \"fd73f227-55d8-4861-9845-1586fd021731\") " Oct 30 13:38:31.702004 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:31.701968 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd73f227-55d8-4861-9845-1586fd021731-maas-api-openshift-ca" pod "fd73f227-55d8-4861-9845-1586fd021731" (UID: "fd73f227-55d8-4861-9845-1586fd021731"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:38:31.702503 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:31.702475 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd73f227-55d8-4861-9845-1586fd021731-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:31.704794 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:31.704750 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd73f227-55d8-4861-9845-1586fd021731-maas-api-xks-ca" pod "fd73f227-55d8-4861-9845-1586fd021731" (UID: "fd73f227-55d8-4861-9845-1586fd021731"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:38:31.705056 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:31.705032 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd73f227-55d8-4861-9845-1586fd021731-kube-api-access-crhv2" pod "fd73f227-55d8-4861-9845-1586fd021731" (UID: "fd73f227-55d8-4861-9845-1586fd021731"). InnerVolumeSpecName "kube-api-access-crhv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:38:31.804043 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:31.803956 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd73f227-55d8-4861-9845-1586fd021731-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:31.804043 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:31.803993 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crhv2\" (UniqueName: \"kubernetes.io/projected/fd73f227-55d8-4861-9845-1586fd021731-kube-api-access-crhv2\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:32.442298 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:32.440888 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c91909-4rb57" Oct 30 13:38:32.442298 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:32.441283 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c91909-4rb57" event={"ID":"fd73f227-55d8-4861-9845-1586fd021731","Type":"ContainerDied","Data":"25a6d8c108cecf2702e01b2bb3ac652e3ddb366793b1858af84ba343eaf82a80"} Oct 30 13:38:32.442298 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:32.441344 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25a6d8c108cecf2702e01b2bb3ac652e3ddb366793b1858af84ba343eaf82a80" Oct 30 13:38:32.459025 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:38:32.458982 4931 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd73f227_55d8_4861_9845_1586fd021731.slice\": RecentStats: unable to find data in memory cache]" Oct 30 13:38:32.661055 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:32.661022 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c8130-674qv"] Oct 30 13:38:32.661754 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:32.661734 4931 cpu_manager.go:414] "containerMap: removing container" podUID="fd73f227-55d8-4861-9845-1586fd021731" containerName="revoke-keys" Oct 30 13:38:32.661754 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:32.661754 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd73f227-55d8-4861-9845-1586fd021731" containerName="revoke-keys" Oct 30 13:38:32.661897 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:32.661854 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd73f227-55d8-4861-9845-1586fd021731" containerName="revoke-keys" Oct 30 13:38:32.665909 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:32.665886 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c8130-674qv" Oct 30 13:38:32.685321 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:32.684029 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c8130-674qv"] Oct 30 13:38:32.712020 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:32.711993 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-mq6wx" Oct 30 13:38:32.793311 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:32.793231 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e71ca0e8-2516-41e8-97b1-71c9be0db7ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1c8130-674qv\" (UID: \"e71ca0e8-2516-41e8-97b1-71c9be0db7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c8130-674qv" Oct 30 13:38:32.793491 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:32.793366 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e71ca0e8-2516-41e8-97b1-71c9be0db7ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1c8130-674qv\" (UID: \"e71ca0e8-2516-41e8-97b1-71c9be0db7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c8130-674qv" Oct 30 13:38:32.793491 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:32.793440 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww4db\" (UniqueName: \"kubernetes.io/projected/e71ca0e8-2516-41e8-97b1-71c9be0db7ee-kube-api-access-ww4db\") pod \"maas-api-revoke-keys-e2e-ipp-b-1c8130-674qv\" (UID: \"e71ca0e8-2516-41e8-97b1-71c9be0db7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c8130-674qv" Oct 30 13:38:32.894725 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:32.894688 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deea9d6c-9b3a-40d5-9b90-74818449aed0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deea9d6c-9b3a-40d5-9b90-74818449aed0-maas-api-openshift-ca\") pod \"deea9d6c-9b3a-40d5-9b90-74818449aed0\" (UID: \"deea9d6c-9b3a-40d5-9b90-74818449aed0\") " Oct 30 13:38:32.894898 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:32.894841 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/deea9d6c-9b3a-40d5-9b90-74818449aed0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deea9d6c-9b3a-40d5-9b90-74818449aed0-maas-api-xks-ca\") pod \"deea9d6c-9b3a-40d5-9b90-74818449aed0\" (UID: \"deea9d6c-9b3a-40d5-9b90-74818449aed0\") " Oct 30 13:38:32.894947 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:32.894937 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deea9d6c-9b3a-40d5-9b90-74818449aed0-kube-api-access-zmqjm\" (UniqueName: \"kubernetes.io/projected/deea9d6c-9b3a-40d5-9b90-74818449aed0-kube-api-access-zmqjm\") pod \"deea9d6c-9b3a-40d5-9b90-74818449aed0\" (UID: \"deea9d6c-9b3a-40d5-9b90-74818449aed0\") " Oct 30 13:38:32.895103 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:32.895074 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deea9d6c-9b3a-40d5-9b90-74818449aed0-maas-api-openshift-ca" pod "deea9d6c-9b3a-40d5-9b90-74818449aed0" (UID: "deea9d6c-9b3a-40d5-9b90-74818449aed0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:38:32.895175 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:32.895154 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e71ca0e8-2516-41e8-97b1-71c9be0db7ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1c8130-674qv\" (UID: \"e71ca0e8-2516-41e8-97b1-71c9be0db7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c8130-674qv" Oct 30 13:38:32.895325 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:32.895306 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww4db\" (UniqueName: \"kubernetes.io/projected/e71ca0e8-2516-41e8-97b1-71c9be0db7ee-kube-api-access-ww4db\") pod \"maas-api-revoke-keys-e2e-ipp-b-1c8130-674qv\" (UID: \"e71ca0e8-2516-41e8-97b1-71c9be0db7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c8130-674qv" Oct 30 13:38:32.895613 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:32.895590 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e71ca0e8-2516-41e8-97b1-71c9be0db7ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1c8130-674qv\" (UID: \"e71ca0e8-2516-41e8-97b1-71c9be0db7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c8130-674qv" Oct 30 13:38:32.895747 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:32.895734 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deea9d6c-9b3a-40d5-9b90-74818449aed0-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:32.896231 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:32.896210 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e71ca0e8-2516-41e8-97b1-71c9be0db7ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1c8130-674qv\" (UID: \"e71ca0e8-2516-41e8-97b1-71c9be0db7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c8130-674qv" Oct 30 13:38:32.897140 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:32.897110 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deea9d6c-9b3a-40d5-9b90-74818449aed0-maas-api-xks-ca" pod "deea9d6c-9b3a-40d5-9b90-74818449aed0" (UID: "deea9d6c-9b3a-40d5-9b90-74818449aed0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:38:32.897301 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:32.897243 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deea9d6c-9b3a-40d5-9b90-74818449aed0-kube-api-access-zmqjm" pod "deea9d6c-9b3a-40d5-9b90-74818449aed0" (UID: "deea9d6c-9b3a-40d5-9b90-74818449aed0"). InnerVolumeSpecName "kube-api-access-zmqjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:38:32.897505 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:32.897485 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e71ca0e8-2516-41e8-97b1-71c9be0db7ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1c8130-674qv\" (UID: \"e71ca0e8-2516-41e8-97b1-71c9be0db7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c8130-674qv" Oct 30 13:38:32.920646 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:32.920611 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww4db\" (UniqueName: \"kubernetes.io/projected/e71ca0e8-2516-41e8-97b1-71c9be0db7ee-kube-api-access-ww4db\") pod \"maas-api-revoke-keys-e2e-ipp-b-1c8130-674qv\" (UID: \"e71ca0e8-2516-41e8-97b1-71c9be0db7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c8130-674qv" Oct 30 13:38:32.997063 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:32.997005 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deea9d6c-9b3a-40d5-9b90-74818449aed0-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:32.997063 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:32.997031 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmqjm\" (UniqueName: \"kubernetes.io/projected/deea9d6c-9b3a-40d5-9b90-74818449aed0-kube-api-access-zmqjm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:33.009918 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:33.009892 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c8130-674qv" Oct 30 13:38:33.324409 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:33.324379 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c8130-674qv"] Oct 30 13:38:33.324535 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:38:33.324507 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71ca0e8_2516_41e8_97b1_71c9be0db7ee.slice/crio-044aacba14c2a5ced1c6c1bf62c2acb8696a7fb32a36ea478b46d33fc8807fc7 WatchSource:0}: Error finding container 044aacba14c2a5ced1c6c1bf62c2acb8696a7fb32a36ea478b46d33fc8807fc7: Status 404 returned error can't find the container with id 044aacba14c2a5ced1c6c1bf62c2acb8696a7fb32a36ea478b46d33fc8807fc7 Oct 30 13:38:33.450242 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:33.450207 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c8130-674qv" event={"ID":"e71ca0e8-2516-41e8-97b1-71c9be0db7ee","Type":"ContainerStarted","Data":"044aacba14c2a5ced1c6c1bf62c2acb8696a7fb32a36ea478b46d33fc8807fc7"} Oct 30 13:38:33.451910 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:33.451883 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-mq6wx" event={"ID":"deea9d6c-9b3a-40d5-9b90-74818449aed0","Type":"ContainerDied","Data":"528cec0087b0443cad1b54b74d5b4c923260f858298d9715be6790c4fac469a0"} Oct 30 13:38:33.452070 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:33.451931 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="528cec0087b0443cad1b54b74d5b4c923260f858298d9715be6790c4fac469a0" Oct 30 13:38:33.452070 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:33.451997 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-mq6wx" Oct 30 13:38:34.348853 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:34.348820 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcd6f9cc-dwvsn"] Oct 30 13:38:34.349508 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:34.349494 4931 cpu_manager.go:414] "containerMap: removing container" podUID="deea9d6c-9b3a-40d5-9b90-74818449aed0" containerName="revoke-keys" Oct 30 13:38:34.349558 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:34.349510 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deea9d6c-9b3a-40d5-9b90-74818449aed0" containerName="revoke-keys" Oct 30 13:38:34.349634 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:34.349625 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="deea9d6c-9b3a-40d5-9b90-74818449aed0" containerName="revoke-keys" Oct 30 13:38:34.353060 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:34.353042 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd6f9cc-dwvsn" Oct 30 13:38:34.379850 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:34.379718 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcd6f9cc-dwvsn"] Oct 30 13:38:34.392022 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:34.391997 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d1153b2-qqh44"] Oct 30 13:38:34.402550 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:34.402522 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1153b2-qqh44" Oct 30 13:38:34.425431 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:34.425357 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d1153b2-qqh44"] Oct 30 13:38:34.462579 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:34.462541 4931 generic.go:356] "Generic (PLEG): container finished" podID="e71ca0e8-2516-41e8-97b1-71c9be0db7ee" containerID="dffb818ff7317219a765512d4566285c60c364c9062fca336f3491bb2a458e3d" exitCode=0 Oct 30 13:38:34.463057 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:34.462627 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c8130-674qv" event={"ID":"e71ca0e8-2516-41e8-97b1-71c9be0db7ee","Type":"ContainerDied","Data":"dffb818ff7317219a765512d4566285c60c364c9062fca336f3491bb2a458e3d"} Oct 30 13:38:34.520118 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:34.520083 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2bc558d-aaa3-4f4d-9834-b4300517b1af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d1153b2-qqh44\" (UID: \"b2bc558d-aaa3-4f4d-9834-b4300517b1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1153b2-qqh44" Oct 30 13:38:34.520320 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:34.520187 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7gpp\" (UniqueName: \"kubernetes.io/projected/b2bc558d-aaa3-4f4d-9834-b4300517b1af-kube-api-access-l7gpp\") pod \"maas-api-revoke-sub-0d1153b2-qqh44\" (UID: \"b2bc558d-aaa3-4f4d-9834-b4300517b1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1153b2-qqh44" Oct 30 13:38:34.520320 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:34.520240 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2bc558d-aaa3-4f4d-9834-b4300517b1af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d1153b2-qqh44\" (UID: \"b2bc558d-aaa3-4f4d-9834-b4300517b1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1153b2-qqh44" Oct 30 13:38:34.520434 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:34.520420 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzg5n\" (UniqueName: \"kubernetes.io/projected/20ca3800-3b1d-46de-a14c-4b9e2da3cd47-kube-api-access-bzg5n\") pod \"maas-api-revoke-sub-dcd6f9cc-dwvsn\" (UID: \"20ca3800-3b1d-46de-a14c-4b9e2da3cd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd6f9cc-dwvsn" Oct 30 13:38:34.520521 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:34.520507 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ca3800-3b1d-46de-a14c-4b9e2da3cd47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcd6f9cc-dwvsn\" (UID: \"20ca3800-3b1d-46de-a14c-4b9e2da3cd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd6f9cc-dwvsn" Oct 30 13:38:34.520559 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:34.520542 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ca3800-3b1d-46de-a14c-4b9e2da3cd47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcd6f9cc-dwvsn\" (UID: \"20ca3800-3b1d-46de-a14c-4b9e2da3cd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd6f9cc-dwvsn" Oct 30 13:38:34.630287 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:34.629316 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2bc558d-aaa3-4f4d-9834-b4300517b1af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d1153b2-qqh44\" (UID: \"b2bc558d-aaa3-4f4d-9834-b4300517b1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1153b2-qqh44" Oct 30 13:38:34.630287 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:34.629416 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7gpp\" (UniqueName: \"kubernetes.io/projected/b2bc558d-aaa3-4f4d-9834-b4300517b1af-kube-api-access-l7gpp\") pod \"maas-api-revoke-sub-0d1153b2-qqh44\" (UID: \"b2bc558d-aaa3-4f4d-9834-b4300517b1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1153b2-qqh44" Oct 30 13:38:34.630287 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:34.629484 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2bc558d-aaa3-4f4d-9834-b4300517b1af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d1153b2-qqh44\" (UID: \"b2bc558d-aaa3-4f4d-9834-b4300517b1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1153b2-qqh44" Oct 30 13:38:34.630287 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:34.629605 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzg5n\" (UniqueName: \"kubernetes.io/projected/20ca3800-3b1d-46de-a14c-4b9e2da3cd47-kube-api-access-bzg5n\") pod \"maas-api-revoke-sub-dcd6f9cc-dwvsn\" (UID: \"20ca3800-3b1d-46de-a14c-4b9e2da3cd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd6f9cc-dwvsn" Oct 30 13:38:34.630287 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:34.629707 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ca3800-3b1d-46de-a14c-4b9e2da3cd47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcd6f9cc-dwvsn\" (UID: \"20ca3800-3b1d-46de-a14c-4b9e2da3cd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd6f9cc-dwvsn" Oct 30 13:38:34.630287 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:34.629764 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ca3800-3b1d-46de-a14c-4b9e2da3cd47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcd6f9cc-dwvsn\" (UID: \"20ca3800-3b1d-46de-a14c-4b9e2da3cd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd6f9cc-dwvsn" Oct 30 13:38:34.633815 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:34.633100 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2bc558d-aaa3-4f4d-9834-b4300517b1af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d1153b2-qqh44\" (UID: \"b2bc558d-aaa3-4f4d-9834-b4300517b1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1153b2-qqh44" Oct 30 13:38:34.635407 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:34.635382 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ca3800-3b1d-46de-a14c-4b9e2da3cd47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcd6f9cc-dwvsn\" (UID: \"20ca3800-3b1d-46de-a14c-4b9e2da3cd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd6f9cc-dwvsn" Oct 30 13:38:34.654045 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:34.653779 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ca3800-3b1d-46de-a14c-4b9e2da3cd47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcd6f9cc-dwvsn\" (UID: \"20ca3800-3b1d-46de-a14c-4b9e2da3cd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd6f9cc-dwvsn" Oct 30 13:38:34.655576 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:34.655546 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2bc558d-aaa3-4f4d-9834-b4300517b1af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d1153b2-qqh44\" (UID: \"b2bc558d-aaa3-4f4d-9834-b4300517b1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1153b2-qqh44" Oct 30 13:38:34.697415 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:34.697382 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7gpp\" (UniqueName: \"kubernetes.io/projected/b2bc558d-aaa3-4f4d-9834-b4300517b1af-kube-api-access-l7gpp\") pod \"maas-api-revoke-sub-0d1153b2-qqh44\" (UID: \"b2bc558d-aaa3-4f4d-9834-b4300517b1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1153b2-qqh44" Oct 30 13:38:34.700107 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:34.700077 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzg5n\" (UniqueName: \"kubernetes.io/projected/20ca3800-3b1d-46de-a14c-4b9e2da3cd47-kube-api-access-bzg5n\") pod \"maas-api-revoke-sub-dcd6f9cc-dwvsn\" (UID: \"20ca3800-3b1d-46de-a14c-4b9e2da3cd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd6f9cc-dwvsn" Oct 30 13:38:34.723975 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:34.723945 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1153b2-qqh44" Oct 30 13:38:34.990345 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:34.980922 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd6f9cc-dwvsn" Oct 30 13:38:35.207287 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:35.204549 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d1153b2-qqh44"] Oct 30 13:38:35.471179 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:35.471145 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1153b2-qqh44" event={"ID":"b2bc558d-aaa3-4f4d-9834-b4300517b1af","Type":"ContainerStarted","Data":"316944a98ac3e1c7170844e3577253b83be1eef1e1a4ea52a3a9b3d51b6f3dc3"} Oct 30 13:38:35.527047 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:35.527019 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcd6f9cc-dwvsn"] Oct 30 13:38:35.640915 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:35.640799 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c8130-674qv" Oct 30 13:38:35.660314 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:35.656652 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e71ca0e8-2516-41e8-97b1-71c9be0db7ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e71ca0e8-2516-41e8-97b1-71c9be0db7ee-maas-api-openshift-ca\") pod \"e71ca0e8-2516-41e8-97b1-71c9be0db7ee\" (UID: \"e71ca0e8-2516-41e8-97b1-71c9be0db7ee\") " Oct 30 13:38:35.660314 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:35.656911 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e71ca0e8-2516-41e8-97b1-71c9be0db7ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e71ca0e8-2516-41e8-97b1-71c9be0db7ee-maas-api-xks-ca\") pod \"e71ca0e8-2516-41e8-97b1-71c9be0db7ee\" (UID: \"e71ca0e8-2516-41e8-97b1-71c9be0db7ee\") " Oct 30 13:38:35.660314 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:35.657173 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e71ca0e8-2516-41e8-97b1-71c9be0db7ee-kube-api-access-ww4db\" (UniqueName: \"kubernetes.io/projected/e71ca0e8-2516-41e8-97b1-71c9be0db7ee-kube-api-access-ww4db\") pod \"e71ca0e8-2516-41e8-97b1-71c9be0db7ee\" (UID: \"e71ca0e8-2516-41e8-97b1-71c9be0db7ee\") " Oct 30 13:38:35.660314 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:35.659450 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e71ca0e8-2516-41e8-97b1-71c9be0db7ee-maas-api-openshift-ca" pod "e71ca0e8-2516-41e8-97b1-71c9be0db7ee" (UID: "e71ca0e8-2516-41e8-97b1-71c9be0db7ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:38:35.692361 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:35.680128 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e71ca0e8-2516-41e8-97b1-71c9be0db7ee-kube-api-access-ww4db" pod "e71ca0e8-2516-41e8-97b1-71c9be0db7ee" (UID: "e71ca0e8-2516-41e8-97b1-71c9be0db7ee"). InnerVolumeSpecName "kube-api-access-ww4db". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:38:35.692361 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:35.680212 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e71ca0e8-2516-41e8-97b1-71c9be0db7ee-maas-api-xks-ca" pod "e71ca0e8-2516-41e8-97b1-71c9be0db7ee" (UID: "e71ca0e8-2516-41e8-97b1-71c9be0db7ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:38:35.759776 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:35.759726 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e71ca0e8-2516-41e8-97b1-71c9be0db7ee-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:35.759776 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:35.759767 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww4db\" (UniqueName: \"kubernetes.io/projected/e71ca0e8-2516-41e8-97b1-71c9be0db7ee-kube-api-access-ww4db\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:35.759776 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:35.759783 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e71ca0e8-2516-41e8-97b1-71c9be0db7ee-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:36.293289 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:36.291236 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-7cxbc"] Oct 30 13:38:36.297101 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:36.297065 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-7cxbc"] Oct 30 13:38:36.479877 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:36.479843 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c8130-674qv" Oct 30 13:38:36.480350 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:36.479850 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c8130-674qv" event={"ID":"e71ca0e8-2516-41e8-97b1-71c9be0db7ee","Type":"ContainerDied","Data":"044aacba14c2a5ced1c6c1bf62c2acb8696a7fb32a36ea478b46d33fc8807fc7"} Oct 30 13:38:36.480350 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:36.479988 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="044aacba14c2a5ced1c6c1bf62c2acb8696a7fb32a36ea478b46d33fc8807fc7" Oct 30 13:38:36.481707 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:36.481684 4931 generic.go:356] "Generic (PLEG): container finished" podID="b2bc558d-aaa3-4f4d-9834-b4300517b1af" containerID="90d1076be300e897f627dd862b517a2adf98fbde22aa9ac8ac68696da7ee2f02" exitCode=0 Oct 30 13:38:36.481812 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:36.481758 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1153b2-qqh44" event={"ID":"b2bc558d-aaa3-4f4d-9834-b4300517b1af","Type":"ContainerDied","Data":"90d1076be300e897f627dd862b517a2adf98fbde22aa9ac8ac68696da7ee2f02"} Oct 30 13:38:36.483706 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:36.483679 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd6f9cc-dwvsn" event={"ID":"20ca3800-3b1d-46de-a14c-4b9e2da3cd47","Type":"ContainerStarted","Data":"5811ca0069e1875201718c5e7485e122c0c641f315c11560311efbf54507453f"} Oct 30 13:38:36.483789 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:36.483728 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd6f9cc-dwvsn" event={"ID":"20ca3800-3b1d-46de-a14c-4b9e2da3cd47","Type":"ContainerStarted","Data":"7c26c0dbc5040df9652e13e861445b46d9cd6170f79996c41eba96a78b3dc85a"} Oct 30 13:38:36.578004 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:36.577954 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd6f9cc-dwvsn" podStartSLOduration=2.577940077 podStartE2EDuration="2.577940077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:38:36.573038233 +0000 UTC m=+3215.176536679" watchObservedRunningTime="2026-10-30 13:38:36.577940077 +0000 UTC m=+3215.181438543" Oct 30 13:38:37.353498 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:38:37.353462 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:38:37.496797 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:37.496760 4931 generic.go:356] "Generic (PLEG): container finished" podID="20ca3800-3b1d-46de-a14c-4b9e2da3cd47" containerID="5811ca0069e1875201718c5e7485e122c0c641f315c11560311efbf54507453f" exitCode=0 Oct 30 13:38:37.497294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:37.496853 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd6f9cc-dwvsn" event={"ID":"20ca3800-3b1d-46de-a14c-4b9e2da3cd47","Type":"ContainerDied","Data":"5811ca0069e1875201718c5e7485e122c0c641f315c11560311efbf54507453f"} Oct 30 13:38:37.516050 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:37.516023 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="607401ec-30ee-476c-bbb8-e3e90c9fec75" path="/var/lib/kubelet/pods/607401ec-30ee-476c-bbb8-e3e90c9fec75/volumes" Oct 30 13:38:37.712514 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:37.712485 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1153b2-qqh44" Oct 30 13:38:37.794076 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:37.794036 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2bc558d-aaa3-4f4d-9834-b4300517b1af-kube-api-access-l7gpp\" (UniqueName: \"kubernetes.io/projected/b2bc558d-aaa3-4f4d-9834-b4300517b1af-kube-api-access-l7gpp\") pod \"b2bc558d-aaa3-4f4d-9834-b4300517b1af\" (UID: \"b2bc558d-aaa3-4f4d-9834-b4300517b1af\") " Oct 30 13:38:37.794249 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:37.794183 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2bc558d-aaa3-4f4d-9834-b4300517b1af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2bc558d-aaa3-4f4d-9834-b4300517b1af-maas-api-xks-ca\") pod \"b2bc558d-aaa3-4f4d-9834-b4300517b1af\" (UID: \"b2bc558d-aaa3-4f4d-9834-b4300517b1af\") " Oct 30 13:38:37.794368 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:37.794349 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2bc558d-aaa3-4f4d-9834-b4300517b1af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2bc558d-aaa3-4f4d-9834-b4300517b1af-maas-api-openshift-ca\") pod \"b2bc558d-aaa3-4f4d-9834-b4300517b1af\" (UID: \"b2bc558d-aaa3-4f4d-9834-b4300517b1af\") " Oct 30 13:38:37.794779 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:37.794734 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2bc558d-aaa3-4f4d-9834-b4300517b1af-maas-api-openshift-ca" pod "b2bc558d-aaa3-4f4d-9834-b4300517b1af" (UID: "b2bc558d-aaa3-4f4d-9834-b4300517b1af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:38:37.795168 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:37.795135 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2bc558d-aaa3-4f4d-9834-b4300517b1af-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:37.797043 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:37.796952 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2bc558d-aaa3-4f4d-9834-b4300517b1af-maas-api-xks-ca" pod "b2bc558d-aaa3-4f4d-9834-b4300517b1af" (UID: "b2bc558d-aaa3-4f4d-9834-b4300517b1af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:38:37.797043 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:37.797009 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2bc558d-aaa3-4f4d-9834-b4300517b1af-kube-api-access-l7gpp" pod "b2bc558d-aaa3-4f4d-9834-b4300517b1af" (UID: "b2bc558d-aaa3-4f4d-9834-b4300517b1af"). InnerVolumeSpecName "kube-api-access-l7gpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:38:37.908007 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:37.906521 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7gpp\" (UniqueName: \"kubernetes.io/projected/b2bc558d-aaa3-4f4d-9834-b4300517b1af-kube-api-access-l7gpp\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:37.908007 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:37.906564 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2bc558d-aaa3-4f4d-9834-b4300517b1af-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:38.505828 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:38.505790 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1153b2-qqh44" Oct 30 13:38:38.505828 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:38.505820 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1153b2-qqh44" event={"ID":"b2bc558d-aaa3-4f4d-9834-b4300517b1af","Type":"ContainerDied","Data":"316944a98ac3e1c7170844e3577253b83be1eef1e1a4ea52a3a9b3d51b6f3dc3"} Oct 30 13:38:38.506325 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:38.505871 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="316944a98ac3e1c7170844e3577253b83be1eef1e1a4ea52a3a9b3d51b6f3dc3" Oct 30 13:38:38.659893 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:38.659869 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd6f9cc-dwvsn" Oct 30 13:38:38.718160 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:38.718123 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20ca3800-3b1d-46de-a14c-4b9e2da3cd47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ca3800-3b1d-46de-a14c-4b9e2da3cd47-maas-api-xks-ca\") pod \"20ca3800-3b1d-46de-a14c-4b9e2da3cd47\" (UID: \"20ca3800-3b1d-46de-a14c-4b9e2da3cd47\") " Oct 30 13:38:38.718386 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:38.718242 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20ca3800-3b1d-46de-a14c-4b9e2da3cd47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ca3800-3b1d-46de-a14c-4b9e2da3cd47-maas-api-openshift-ca\") pod \"20ca3800-3b1d-46de-a14c-4b9e2da3cd47\" (UID: \"20ca3800-3b1d-46de-a14c-4b9e2da3cd47\") " Oct 30 13:38:38.718386 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:38.718344 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20ca3800-3b1d-46de-a14c-4b9e2da3cd47-kube-api-access-bzg5n\" (UniqueName: \"kubernetes.io/projected/20ca3800-3b1d-46de-a14c-4b9e2da3cd47-kube-api-access-bzg5n\") pod \"20ca3800-3b1d-46de-a14c-4b9e2da3cd47\" (UID: \"20ca3800-3b1d-46de-a14c-4b9e2da3cd47\") " Oct 30 13:38:38.718671 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:38.718641 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20ca3800-3b1d-46de-a14c-4b9e2da3cd47-maas-api-openshift-ca" pod "20ca3800-3b1d-46de-a14c-4b9e2da3cd47" (UID: "20ca3800-3b1d-46de-a14c-4b9e2da3cd47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:38:38.719056 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:38.719032 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ca3800-3b1d-46de-a14c-4b9e2da3cd47-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:38.720562 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:38.720535 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20ca3800-3b1d-46de-a14c-4b9e2da3cd47-maas-api-xks-ca" pod "20ca3800-3b1d-46de-a14c-4b9e2da3cd47" (UID: "20ca3800-3b1d-46de-a14c-4b9e2da3cd47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:38:38.720738 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:38.720721 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20ca3800-3b1d-46de-a14c-4b9e2da3cd47-kube-api-access-bzg5n" pod "20ca3800-3b1d-46de-a14c-4b9e2da3cd47" (UID: "20ca3800-3b1d-46de-a14c-4b9e2da3cd47"). InnerVolumeSpecName "kube-api-access-bzg5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:38:38.820144 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:38.820069 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ca3800-3b1d-46de-a14c-4b9e2da3cd47-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:38.820144 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:38.820100 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzg5n\" (UniqueName: \"kubernetes.io/projected/20ca3800-3b1d-46de-a14c-4b9e2da3cd47-kube-api-access-bzg5n\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:39.515468 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:39.515440 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd6f9cc-dwvsn" Oct 30 13:38:39.515949 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:39.515435 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd6f9cc-dwvsn" event={"ID":"20ca3800-3b1d-46de-a14c-4b9e2da3cd47","Type":"ContainerDied","Data":"7c26c0dbc5040df9652e13e861445b46d9cd6170f79996c41eba96a78b3dc85a"} Oct 30 13:38:39.515949 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:39.515561 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c26c0dbc5040df9652e13e861445b46d9cd6170f79996c41eba96a78b3dc85a" Oct 30 13:38:43.487591 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:43.487544 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn"] Oct 30 13:38:43.488120 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:43.487815 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" podUID="b446da63-fda1-43e4-91b0-9a2d2a77b1a1" containerName="istio-proxy" containerID="cri-o://f058bc5026bf2957d921ed6b72ac80ae5973e89a4e3f4473db01c8d4d0954853" gracePeriod=30 Oct 30 13:38:45.405435 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:45.405399 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-96ab73-qxttd"] Oct 30 13:38:45.406311 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:45.406256 4931 cpu_manager.go:414] "containerMap: removing container" podUID="b2bc558d-aaa3-4f4d-9834-b4300517b1af" containerName="revoke-keys" Oct 30 13:38:45.406311 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:45.406293 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2bc558d-aaa3-4f4d-9834-b4300517b1af" containerName="revoke-keys" Oct 30 13:38:45.406311 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:45.406314 4931 cpu_manager.go:414] "containerMap: removing container" podUID="e71ca0e8-2516-41e8-97b1-71c9be0db7ee" containerName="revoke-keys" Oct 30 13:38:45.406583 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:45.406322 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e71ca0e8-2516-41e8-97b1-71c9be0db7ee" containerName="revoke-keys" Oct 30 13:38:45.406583 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:45.406331 4931 cpu_manager.go:414] "containerMap: removing container" podUID="20ca3800-3b1d-46de-a14c-4b9e2da3cd47" containerName="revoke-keys" Oct 30 13:38:45.406583 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:45.406339 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20ca3800-3b1d-46de-a14c-4b9e2da3cd47" containerName="revoke-keys" Oct 30 13:38:45.406583 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:45.406510 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2bc558d-aaa3-4f4d-9834-b4300517b1af" containerName="revoke-keys" Oct 30 13:38:45.406583 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:45.406528 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="e71ca0e8-2516-41e8-97b1-71c9be0db7ee" containerName="revoke-keys" Oct 30 13:38:45.406583 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:45.406540 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="20ca3800-3b1d-46de-a14c-4b9e2da3cd47" containerName="revoke-keys" Oct 30 13:38:45.410557 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:45.410535 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-96ab73-qxttd" Oct 30 13:38:45.430058 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:45.430034 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-96ab73-qxttd"] Oct 30 13:38:45.437802 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:45.437777 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:38:45.437995 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:45.437869 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:38:45.555307 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:45.555259 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxm4k\" (UniqueName: \"kubernetes.io/projected/66b6003e-28c2-4275-9b9c-6865fd5f7b60-kube-api-access-zxm4k\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-96ab73-qxttd\" (UID: \"66b6003e-28c2-4275-9b9c-6865fd5f7b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-96ab73-qxttd" Oct 30 13:38:45.555463 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:45.555441 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66b6003e-28c2-4275-9b9c-6865fd5f7b60-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-96ab73-qxttd\" (UID: \"66b6003e-28c2-4275-9b9c-6865fd5f7b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-96ab73-qxttd" Oct 30 13:38:45.555534 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:45.555502 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66b6003e-28c2-4275-9b9c-6865fd5f7b60-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-96ab73-qxttd\" (UID: \"66b6003e-28c2-4275-9b9c-6865fd5f7b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-96ab73-qxttd" Oct 30 13:38:45.657357 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:45.657245 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxm4k\" (UniqueName: \"kubernetes.io/projected/66b6003e-28c2-4275-9b9c-6865fd5f7b60-kube-api-access-zxm4k\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-96ab73-qxttd\" (UID: \"66b6003e-28c2-4275-9b9c-6865fd5f7b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-96ab73-qxttd" Oct 30 13:38:45.657562 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:45.657417 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66b6003e-28c2-4275-9b9c-6865fd5f7b60-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-96ab73-qxttd\" (UID: \"66b6003e-28c2-4275-9b9c-6865fd5f7b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-96ab73-qxttd" Oct 30 13:38:45.657562 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:45.657500 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66b6003e-28c2-4275-9b9c-6865fd5f7b60-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-96ab73-qxttd\" (UID: \"66b6003e-28c2-4275-9b9c-6865fd5f7b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-96ab73-qxttd" Oct 30 13:38:45.658207 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:45.658183 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66b6003e-28c2-4275-9b9c-6865fd5f7b60-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-96ab73-qxttd\" (UID: \"66b6003e-28c2-4275-9b9c-6865fd5f7b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-96ab73-qxttd" Oct 30 13:38:45.660121 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:45.660092 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66b6003e-28c2-4275-9b9c-6865fd5f7b60-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-96ab73-qxttd\" (UID: \"66b6003e-28c2-4275-9b9c-6865fd5f7b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-96ab73-qxttd" Oct 30 13:38:45.683764 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:45.683739 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxm4k\" (UniqueName: \"kubernetes.io/projected/66b6003e-28c2-4275-9b9c-6865fd5f7b60-kube-api-access-zxm4k\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-96ab73-qxttd\" (UID: \"66b6003e-28c2-4275-9b9c-6865fd5f7b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-96ab73-qxttd" Oct 30 13:38:45.725690 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:45.725661 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-96ab73-qxttd" Oct 30 13:38:46.353663 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:46.353631 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-96ab73-qxttd"] Oct 30 13:38:46.573557 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:46.573524 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-96ab73-qxttd" event={"ID":"66b6003e-28c2-4275-9b9c-6865fd5f7b60","Type":"ContainerStarted","Data":"e05e3cf8b4ed689ef4358b479c149d731d1f2540239a52788127a95c38df248f"} Oct 30 13:38:46.925300 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:46.925153 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-srqbd"] Oct 30 13:38:46.938361 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:46.938321 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-srqbd"] Oct 30 13:38:47.514940 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:47.514904 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6141898a-d732-491e-aca2-ad76f1acb683" path="/var/lib/kubelet/pods/6141898a-d732-491e-aca2-ad76f1acb683/volumes" Oct 30 13:38:47.582385 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:47.582351 4931 generic.go:356] "Generic (PLEG): container finished" podID="66b6003e-28c2-4275-9b9c-6865fd5f7b60" containerID="31517084b34d3d04199f74fcd5550344b8be3bb3b1810c632f553503c31ebc4d" exitCode=0 Oct 30 13:38:47.582758 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:47.582400 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-96ab73-qxttd" event={"ID":"66b6003e-28c2-4275-9b9c-6865fd5f7b60","Type":"ContainerDied","Data":"31517084b34d3d04199f74fcd5550344b8be3bb3b1810c632f553503c31ebc4d"} Oct 30 13:38:48.774082 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.774058 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-96ab73-qxttd" Oct 30 13:38:48.832144 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.832121 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:38:48.886350 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.886315 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-kube-api-access-hjk9z\" (UniqueName: \"kubernetes.io/projected/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-kube-api-access-hjk9z\") pod \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " Oct 30 13:38:48.886550 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.886371 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-istio-envoy\") pod \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " Oct 30 13:38:48.886550 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.886425 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-istio-podinfo\") pod \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " Oct 30 13:38:48.886550 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.886493 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66b6003e-28c2-4275-9b9c-6865fd5f7b60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66b6003e-28c2-4275-9b9c-6865fd5f7b60-maas-api-xks-ca\") pod \"66b6003e-28c2-4275-9b9c-6865fd5f7b60\" (UID: \"66b6003e-28c2-4275-9b9c-6865fd5f7b60\") " Oct 30 13:38:48.886821 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.886550 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-istiod-ca-cert\") pod \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " Oct 30 13:38:48.886821 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.886583 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-workload-socket\") pod \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " Oct 30 13:38:48.886821 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.886647 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66b6003e-28c2-4275-9b9c-6865fd5f7b60-kube-api-access-zxm4k\" (UniqueName: \"kubernetes.io/projected/66b6003e-28c2-4275-9b9c-6865fd5f7b60-kube-api-access-zxm4k\") pod \"66b6003e-28c2-4275-9b9c-6865fd5f7b60\" (UID: \"66b6003e-28c2-4275-9b9c-6865fd5f7b60\") " Oct 30 13:38:48.886821 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.886732 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-workload-certs\") pod \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " Oct 30 13:38:48.886821 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.886800 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-istio-token\" (UniqueName: \"kubernetes.io/projected/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-istio-token\") pod \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " Oct 30 13:38:48.887108 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.886916 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-istio-data\") pod \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " Oct 30 13:38:48.887108 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.886964 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-workload-socket" pod "b446da63-fda1-43e4-91b0-9a2d2a77b1a1" (UID: "b446da63-fda1-43e4-91b0-9a2d2a77b1a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:38:48.887108 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.886971 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66b6003e-28c2-4275-9b9c-6865fd5f7b60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66b6003e-28c2-4275-9b9c-6865fd5f7b60-maas-api-openshift-ca\") pod \"66b6003e-28c2-4275-9b9c-6865fd5f7b60\" (UID: \"66b6003e-28c2-4275-9b9c-6865fd5f7b60\") " Oct 30 13:38:48.887108 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.887037 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-credential-socket\") pod \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\" (UID: \"b446da63-fda1-43e4-91b0-9a2d2a77b1a1\") " Oct 30 13:38:48.887334 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.887291 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66b6003e-28c2-4275-9b9c-6865fd5f7b60-maas-api-openshift-ca" pod "66b6003e-28c2-4275-9b9c-6865fd5f7b60" (UID: "66b6003e-28c2-4275-9b9c-6865fd5f7b60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:38:48.888122 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.887504 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-workload-certs" pod "b446da63-fda1-43e4-91b0-9a2d2a77b1a1" (UID: "b446da63-fda1-43e4-91b0-9a2d2a77b1a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:38:48.888122 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.887575 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-istio-data" pod "b446da63-fda1-43e4-91b0-9a2d2a77b1a1" (UID: "b446da63-fda1-43e4-91b0-9a2d2a77b1a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:38:48.888122 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.887711 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-istiod-ca-cert" pod "b446da63-fda1-43e4-91b0-9a2d2a77b1a1" (UID: "b446da63-fda1-43e4-91b0-9a2d2a77b1a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:38:48.888455 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.888136 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-credential-socket" pod "b446da63-fda1-43e4-91b0-9a2d2a77b1a1" (UID: "b446da63-fda1-43e4-91b0-9a2d2a77b1a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:38:48.888826 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.888805 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:48.888826 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.888831 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:48.888972 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.888845 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:48.888972 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.888858 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:48.888972 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.888870 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66b6003e-28c2-4275-9b9c-6865fd5f7b60-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:48.888972 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.888882 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:48.890069 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.890032 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-kube-api-access-hjk9z" pod "b446da63-fda1-43e4-91b0-9a2d2a77b1a1" (UID: "b446da63-fda1-43e4-91b0-9a2d2a77b1a1"). InnerVolumeSpecName "kube-api-access-hjk9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:38:48.890230 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.890083 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-istio-envoy" pod "b446da63-fda1-43e4-91b0-9a2d2a77b1a1" (UID: "b446da63-fda1-43e4-91b0-9a2d2a77b1a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:38:48.890230 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.890206 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-istio-token" pod "b446da63-fda1-43e4-91b0-9a2d2a77b1a1" (UID: "b446da63-fda1-43e4-91b0-9a2d2a77b1a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:38:48.890453 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.890434 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66b6003e-28c2-4275-9b9c-6865fd5f7b60-maas-api-xks-ca" pod "66b6003e-28c2-4275-9b9c-6865fd5f7b60" (UID: "66b6003e-28c2-4275-9b9c-6865fd5f7b60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:38:48.891427 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.891406 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66b6003e-28c2-4275-9b9c-6865fd5f7b60-kube-api-access-zxm4k" pod "66b6003e-28c2-4275-9b9c-6865fd5f7b60" (UID: "66b6003e-28c2-4275-9b9c-6865fd5f7b60"). InnerVolumeSpecName "kube-api-access-zxm4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:38:48.891991 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.891970 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-istio-podinfo" pod "b446da63-fda1-43e4-91b0-9a2d2a77b1a1" (UID: "b446da63-fda1-43e4-91b0-9a2d2a77b1a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:38:48.990799 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.990758 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:48.990799 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.990795 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66b6003e-28c2-4275-9b9c-6865fd5f7b60-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:48.990799 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.990806 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxm4k\" (UniqueName: \"kubernetes.io/projected/66b6003e-28c2-4275-9b9c-6865fd5f7b60-kube-api-access-zxm4k\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:48.991140 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.990817 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:48.991140 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.990826 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjk9z\" (UniqueName: \"kubernetes.io/projected/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-kube-api-access-hjk9z\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:48.991140 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:48.990835 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b446da63-fda1-43e4-91b0-9a2d2a77b1a1-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:38:49.600399 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:49.600364 4931 generic.go:356] "Generic (PLEG): container finished" podID="b446da63-fda1-43e4-91b0-9a2d2a77b1a1" containerID="f058bc5026bf2957d921ed6b72ac80ae5973e89a4e3f4473db01c8d4d0954853" exitCode=0 Oct 30 13:38:49.600575 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:49.600431 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" Oct 30 13:38:49.600575 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:49.600447 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" event={"ID":"b446da63-fda1-43e4-91b0-9a2d2a77b1a1","Type":"ContainerDied","Data":"f058bc5026bf2957d921ed6b72ac80ae5973e89a4e3f4473db01c8d4d0954853"} Oct 30 13:38:49.600575 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:49.600510 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn" event={"ID":"b446da63-fda1-43e4-91b0-9a2d2a77b1a1","Type":"ContainerDied","Data":"312b7f83ff9d992297cbf7a075078661e5b575456f437ec507ceceafe92530ea"} Oct 30 13:38:49.600575 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:49.600528 4931 scope.go:122] "RemoveContainer" containerID="f058bc5026bf2957d921ed6b72ac80ae5973e89a4e3f4473db01c8d4d0954853" Oct 30 13:38:49.602240 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:49.602182 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-96ab73-qxttd" Oct 30 13:38:49.602240 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:49.602208 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-96ab73-qxttd" event={"ID":"66b6003e-28c2-4275-9b9c-6865fd5f7b60","Type":"ContainerDied","Data":"e05e3cf8b4ed689ef4358b479c149d731d1f2540239a52788127a95c38df248f"} Oct 30 13:38:49.602418 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:49.602318 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e05e3cf8b4ed689ef4358b479c149d731d1f2540239a52788127a95c38df248f" Oct 30 13:38:49.612627 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:49.612607 4931 scope.go:122] "RemoveContainer" containerID="ceefc64030e945fa07b7d162f07b292ce681a38af7d32cce715606702a3af29d" Oct 30 13:38:49.624312 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:49.624259 4931 scope.go:122] "RemoveContainer" containerID="f058bc5026bf2957d921ed6b72ac80ae5973e89a4e3f4473db01c8d4d0954853" Oct 30 13:38:49.624630 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:38:49.624606 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f058bc5026bf2957d921ed6b72ac80ae5973e89a4e3f4473db01c8d4d0954853\": container with ID starting with f058bc5026bf2957d921ed6b72ac80ae5973e89a4e3f4473db01c8d4d0954853 not found: ID does not exist" containerID="f058bc5026bf2957d921ed6b72ac80ae5973e89a4e3f4473db01c8d4d0954853" Oct 30 13:38:49.624736 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:49.624636 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f058bc5026bf2957d921ed6b72ac80ae5973e89a4e3f4473db01c8d4d0954853"} err="failed to get container status \"f058bc5026bf2957d921ed6b72ac80ae5973e89a4e3f4473db01c8d4d0954853\": rpc error: code = NotFound desc = could not find container \"f058bc5026bf2957d921ed6b72ac80ae5973e89a4e3f4473db01c8d4d0954853\": container with ID starting with f058bc5026bf2957d921ed6b72ac80ae5973e89a4e3f4473db01c8d4d0954853 not found: ID does not exist" Oct 30 13:38:49.624736 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:49.624653 4931 scope.go:122] "RemoveContainer" containerID="ceefc64030e945fa07b7d162f07b292ce681a38af7d32cce715606702a3af29d" Oct 30 13:38:49.624904 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:38:49.624885 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceefc64030e945fa07b7d162f07b292ce681a38af7d32cce715606702a3af29d\": container with ID starting with ceefc64030e945fa07b7d162f07b292ce681a38af7d32cce715606702a3af29d not found: ID does not exist" containerID="ceefc64030e945fa07b7d162f07b292ce681a38af7d32cce715606702a3af29d" Oct 30 13:38:49.624969 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:49.624909 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceefc64030e945fa07b7d162f07b292ce681a38af7d32cce715606702a3af29d"} err="failed to get container status \"ceefc64030e945fa07b7d162f07b292ce681a38af7d32cce715606702a3af29d\": rpc error: code = NotFound desc = could not find container \"ceefc64030e945fa07b7d162f07b292ce681a38af7d32cce715606702a3af29d\": container with ID starting with ceefc64030e945fa07b7d162f07b292ce681a38af7d32cce715606702a3af29d not found: ID does not exist" Oct 30 13:38:49.678010 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:49.677980 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn"] Oct 30 13:38:49.691144 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:49.691110 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn"] Oct 30 13:38:51.646238 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:51.646183 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b446da63-fda1-43e4-91b0-9a2d2a77b1a1" path="/var/lib/kubelet/pods/b446da63-fda1-43e4-91b0-9a2d2a77b1a1/volumes" Oct 30 13:38:51.646782 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:51.646754 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c8130-674qv"] Oct 30 13:38:51.646856 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:51.646794 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c8130-674qv"] Oct 30 13:38:51.805815 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:51.805771 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:38:53.515310 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:53.515259 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e71ca0e8-2516-41e8-97b1-71c9be0db7ee" path="/var/lib/kubelet/pods/e71ca0e8-2516-41e8-97b1-71c9be0db7ee/volumes" Oct 30 13:38:54.699067 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:54.699029 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj"] Oct 30 13:38:54.699588 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:54.699230 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" podUID="c666cfb4-8b66-49b3-b248-2f2771088444" containerName="istio-proxy" containerID="cri-o://b1fbfd0a3711a27de8b03930069de51ec828290cf14a6961383d360484d02453" gracePeriod=30 Oct 30 13:38:54.761055 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:54.761021 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd"] Oct 30 13:38:54.761245 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:54.761223 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" podUID="6a38c538-ca9f-4135-96f8-32dff99e07d7" containerName="istio-proxy" containerID="cri-o://094e0849ae68f5e03c375b4456a2242828b445abd1bc0e6d277926a5dcce6e3c" gracePeriod=30 Oct 30 13:38:55.024063 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:55.023255 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg"] Oct 30 13:38:55.024696 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:55.024640 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" podUID="5c6dbc7e-8785-4971-88c4-e3341faaadff" containerName="istio-proxy" containerID="cri-o://9148f6cf302e45329dca322c30a067837c39207a4d11f48b7cb56acb3493f713" gracePeriod=30 Oct 30 13:38:56.843717 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:56.842296 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-acd125-w5qhp"] Oct 30 13:38:56.843717 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:56.843596 4931 cpu_manager.go:414] "containerMap: removing container" podUID="b446da63-fda1-43e4-91b0-9a2d2a77b1a1" containerName="istio-proxy" Oct 30 13:38:56.843717 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:56.843620 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b446da63-fda1-43e4-91b0-9a2d2a77b1a1" containerName="istio-proxy" Oct 30 13:38:56.843717 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:56.843644 4931 cpu_manager.go:414] "containerMap: removing container" podUID="b446da63-fda1-43e4-91b0-9a2d2a77b1a1" containerName="istio-proxy" Oct 30 13:38:56.843717 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:56.843654 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b446da63-fda1-43e4-91b0-9a2d2a77b1a1" containerName="istio-proxy" Oct 30 13:38:56.843717 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:56.843690 4931 cpu_manager.go:414] "containerMap: removing container" podUID="66b6003e-28c2-4275-9b9c-6865fd5f7b60" containerName="revoke-keys" Oct 30 13:38:56.843717 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:56.843699 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66b6003e-28c2-4275-9b9c-6865fd5f7b60" containerName="revoke-keys" Oct 30 13:38:56.844846 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:56.843834 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b446da63-fda1-43e4-91b0-9a2d2a77b1a1" containerName="istio-proxy" Oct 30 13:38:56.844846 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:56.843853 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b446da63-fda1-43e4-91b0-9a2d2a77b1a1" containerName="istio-proxy" Oct 30 13:38:56.844846 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:56.843869 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="66b6003e-28c2-4275-9b9c-6865fd5f7b60" containerName="revoke-keys" Oct 30 13:38:56.850344 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:56.849987 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-acd125-w5qhp" Oct 30 13:38:56.875052 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:56.875005 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-acd125-w5qhp"] Oct 30 13:38:56.879389 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:56.879341 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:38:56.879564 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:56.879440 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:38:56.922285 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:56.922228 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78h86\" (UniqueName: \"kubernetes.io/projected/89b2ce06-3adf-4653-b527-2b1bf252cde6-kube-api-access-78h86\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-acd125-w5qhp\" (UID: \"89b2ce06-3adf-4653-b527-2b1bf252cde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-acd125-w5qhp" Oct 30 13:38:56.922469 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:56.922321 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89b2ce06-3adf-4653-b527-2b1bf252cde6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-acd125-w5qhp\" (UID: \"89b2ce06-3adf-4653-b527-2b1bf252cde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-acd125-w5qhp" Oct 30 13:38:56.922469 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:56.922397 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89b2ce06-3adf-4653-b527-2b1bf252cde6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-acd125-w5qhp\" (UID: \"89b2ce06-3adf-4653-b527-2b1bf252cde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-acd125-w5qhp" Oct 30 13:38:57.023740 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:57.023699 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78h86\" (UniqueName: \"kubernetes.io/projected/89b2ce06-3adf-4653-b527-2b1bf252cde6-kube-api-access-78h86\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-acd125-w5qhp\" (UID: \"89b2ce06-3adf-4653-b527-2b1bf252cde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-acd125-w5qhp" Oct 30 13:38:57.023963 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:57.023763 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89b2ce06-3adf-4653-b527-2b1bf252cde6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-acd125-w5qhp\" (UID: \"89b2ce06-3adf-4653-b527-2b1bf252cde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-acd125-w5qhp" Oct 30 13:38:57.023963 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:57.023786 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89b2ce06-3adf-4653-b527-2b1bf252cde6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-acd125-w5qhp\" (UID: \"89b2ce06-3adf-4653-b527-2b1bf252cde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-acd125-w5qhp" Oct 30 13:38:57.024502 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:57.024476 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89b2ce06-3adf-4653-b527-2b1bf252cde6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-acd125-w5qhp\" (UID: \"89b2ce06-3adf-4653-b527-2b1bf252cde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-acd125-w5qhp" Oct 30 13:38:57.026214 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:57.026192 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89b2ce06-3adf-4653-b527-2b1bf252cde6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-acd125-w5qhp\" (UID: \"89b2ce06-3adf-4653-b527-2b1bf252cde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-acd125-w5qhp" Oct 30 13:38:57.048540 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:57.048506 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78h86\" (UniqueName: \"kubernetes.io/projected/89b2ce06-3adf-4653-b527-2b1bf252cde6-kube-api-access-78h86\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-acd125-w5qhp\" (UID: \"89b2ce06-3adf-4653-b527-2b1bf252cde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-acd125-w5qhp" Oct 30 13:38:57.171494 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:57.171410 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-acd125-w5qhp" Oct 30 13:38:57.495397 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:38:57.495365 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b2ce06_3adf_4653_b527_2b1bf252cde6.slice/crio-f0bd45471551fb218c523d69602313f1855aa042683004738bc39ee8ccf171e9 WatchSource:0}: Error finding container f0bd45471551fb218c523d69602313f1855aa042683004738bc39ee8ccf171e9: Status 404 returned error can't find the container with id f0bd45471551fb218c523d69602313f1855aa042683004738bc39ee8ccf171e9 Oct 30 13:38:57.495762 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:57.495739 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-acd125-w5qhp"] Oct 30 13:38:57.741570 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:57.741482 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-acd125-w5qhp" event={"ID":"89b2ce06-3adf-4653-b527-2b1bf252cde6","Type":"ContainerStarted","Data":"f0bd45471551fb218c523d69602313f1855aa042683004738bc39ee8ccf171e9"} Oct 30 13:38:58.750304 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:58.750245 4931 generic.go:356] "Generic (PLEG): container finished" podID="89b2ce06-3adf-4653-b527-2b1bf252cde6" containerID="cb7e2ca00c4b75e7bcdffdda4369223cb64ec81bc8a4200dafc8eca887e54902" exitCode=0 Oct 30 13:38:58.750698 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:58.750333 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-acd125-w5qhp" event={"ID":"89b2ce06-3adf-4653-b527-2b1bf252cde6","Type":"ContainerDied","Data":"cb7e2ca00c4b75e7bcdffdda4369223cb64ec81bc8a4200dafc8eca887e54902"} Oct 30 13:38:58.860583 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:58.860496 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm"] Oct 30 13:38:58.860779 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:58.860710 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" podUID="813ffc2b-3c94-4847-aa9b-2d7971da6e4d" containerName="istio-proxy" containerID="cri-o://5498caf666415906672c692ac63dd600423c82d515682a426064d18b60c9fcb7" gracePeriod=30 Oct 30 13:38:58.867891 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:58.867860 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z"] Oct 30 13:38:58.868108 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:58.868080 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" podUID="9916ba3d-3bef-4338-aa2e-f5c27782188f" containerName="istio-proxy" containerID="cri-o://84b32dc6f16656f85128e413f1bc9741ef177676729f2f0ba6c25a4ff550df82" gracePeriod=30 Oct 30 13:38:58.871761 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:58.871732 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p"] Oct 30 13:38:58.871967 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:58.871925 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" podUID="d6f71a7e-fbeb-4e16-8863-e17fc9695553" containerName="istio-proxy" containerID="cri-o://0ab016e7e6e72b08b3a2959061f4d839b39b1e4db4d5db283b4375e744948a45" gracePeriod=30 Oct 30 13:38:59.393876 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:59.393834 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2afe58-g92hc"] Oct 30 13:38:59.398510 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:59.398486 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2afe58-g92hc" Oct 30 13:38:59.420844 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:59.420822 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2afe58-g92hc"] Oct 30 13:38:59.427851 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:59.427806 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ba37aa6-09b2-4e31-a7f4-86db3413c202-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2afe58-g92hc\" (UID: \"6ba37aa6-09b2-4e31-a7f4-86db3413c202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2afe58-g92hc" Oct 30 13:38:59.428117 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:59.428093 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv5pd\" (UniqueName: \"kubernetes.io/projected/6ba37aa6-09b2-4e31-a7f4-86db3413c202-kube-api-access-jv5pd\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2afe58-g92hc\" (UID: \"6ba37aa6-09b2-4e31-a7f4-86db3413c202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2afe58-g92hc" Oct 30 13:38:59.428344 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:59.428328 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ba37aa6-09b2-4e31-a7f4-86db3413c202-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2afe58-g92hc\" (UID: \"6ba37aa6-09b2-4e31-a7f4-86db3413c202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2afe58-g92hc" Oct 30 13:38:59.531238 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:59.531196 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv5pd\" (UniqueName: \"kubernetes.io/projected/6ba37aa6-09b2-4e31-a7f4-86db3413c202-kube-api-access-jv5pd\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2afe58-g92hc\" (UID: \"6ba37aa6-09b2-4e31-a7f4-86db3413c202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2afe58-g92hc" Oct 30 13:38:59.531458 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:59.531308 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ba37aa6-09b2-4e31-a7f4-86db3413c202-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2afe58-g92hc\" (UID: \"6ba37aa6-09b2-4e31-a7f4-86db3413c202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2afe58-g92hc" Oct 30 13:38:59.531458 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:59.531445 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ba37aa6-09b2-4e31-a7f4-86db3413c202-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2afe58-g92hc\" (UID: \"6ba37aa6-09b2-4e31-a7f4-86db3413c202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2afe58-g92hc" Oct 30 13:38:59.531961 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:59.531934 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ba37aa6-09b2-4e31-a7f4-86db3413c202-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2afe58-g92hc\" (UID: \"6ba37aa6-09b2-4e31-a7f4-86db3413c202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2afe58-g92hc" Oct 30 13:38:59.533900 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:59.533875 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ba37aa6-09b2-4e31-a7f4-86db3413c202-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2afe58-g92hc\" (UID: \"6ba37aa6-09b2-4e31-a7f4-86db3413c202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2afe58-g92hc" Oct 30 13:38:59.555943 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:59.555919 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv5pd\" (UniqueName: \"kubernetes.io/projected/6ba37aa6-09b2-4e31-a7f4-86db3413c202-kube-api-access-jv5pd\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2afe58-g92hc\" (UID: \"6ba37aa6-09b2-4e31-a7f4-86db3413c202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2afe58-g92hc" Oct 30 13:38:59.629469 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:59.629440 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:38:59.629591 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:59.629493 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" podUID="c666cfb4-8b66-49b3-b248-2f2771088444" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:38:59.713402 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:59.713375 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2afe58-g92hc" Oct 30 13:38:59.887459 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:59.887423 4931 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 30 13:38:59.887890 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:59.887479 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" podUID="6a38c538-ca9f-4135-96f8-32dff99e07d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 30 13:38:59.990823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:38:59.990490 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-acd125-w5qhp" Oct 30 13:39:00.048362 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:39:00.038305 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ba37aa6_09b2_4e31_a7f4_86db3413c202.slice/crio-882fb94a6ddaf2cca9a93e53ff3416e3500998e47998c88563a6543d3dd8a6fb WatchSource:0}: Error finding container 882fb94a6ddaf2cca9a93e53ff3416e3500998e47998c88563a6543d3dd8a6fb: Status 404 returned error can't find the container with id 882fb94a6ddaf2cca9a93e53ff3416e3500998e47998c88563a6543d3dd8a6fb Oct 30 13:39:00.048362 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.039781 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89b2ce06-3adf-4653-b527-2b1bf252cde6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89b2ce06-3adf-4653-b527-2b1bf252cde6-maas-api-openshift-ca\") pod \"89b2ce06-3adf-4653-b527-2b1bf252cde6\" (UID: \"89b2ce06-3adf-4653-b527-2b1bf252cde6\") " Oct 30 13:39:00.048362 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.039927 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b2ce06-3adf-4653-b527-2b1bf252cde6-kube-api-access-78h86\" (UniqueName: \"kubernetes.io/projected/89b2ce06-3adf-4653-b527-2b1bf252cde6-kube-api-access-78h86\") pod \"89b2ce06-3adf-4653-b527-2b1bf252cde6\" (UID: \"89b2ce06-3adf-4653-b527-2b1bf252cde6\") " Oct 30 13:39:00.048362 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.040029 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89b2ce06-3adf-4653-b527-2b1bf252cde6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89b2ce06-3adf-4653-b527-2b1bf252cde6-maas-api-xks-ca\") pod \"89b2ce06-3adf-4653-b527-2b1bf252cde6\" (UID: \"89b2ce06-3adf-4653-b527-2b1bf252cde6\") " Oct 30 13:39:00.048362 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.041097 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b2ce06-3adf-4653-b527-2b1bf252cde6-maas-api-openshift-ca" pod "89b2ce06-3adf-4653-b527-2b1bf252cde6" (UID: "89b2ce06-3adf-4653-b527-2b1bf252cde6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:39:00.048362 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.043334 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2afe58-g92hc"] Oct 30 13:39:00.063101 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.063070 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89b2ce06-3adf-4653-b527-2b1bf252cde6-maas-api-xks-ca" pod "89b2ce06-3adf-4653-b527-2b1bf252cde6" (UID: "89b2ce06-3adf-4653-b527-2b1bf252cde6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:39:00.065848 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.065812 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b2ce06-3adf-4653-b527-2b1bf252cde6-kube-api-access-78h86" pod "89b2ce06-3adf-4653-b527-2b1bf252cde6" (UID: "89b2ce06-3adf-4653-b527-2b1bf252cde6"). InnerVolumeSpecName "kube-api-access-78h86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:00.143465 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.142857 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89b2ce06-3adf-4653-b527-2b1bf252cde6-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:00.143465 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.142887 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78h86\" (UniqueName: \"kubernetes.io/projected/89b2ce06-3adf-4653-b527-2b1bf252cde6-kube-api-access-78h86\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:00.143465 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.142903 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89b2ce06-3adf-4653-b527-2b1bf252cde6-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:00.146779 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.146482 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" Oct 30 13:39:00.163733 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.163702 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" Oct 30 13:39:00.246927 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.246849 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a38c538-ca9f-4135-96f8-32dff99e07d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a38c538-ca9f-4135-96f8-32dff99e07d7-istio-podinfo\") pod \"6a38c538-ca9f-4135-96f8-32dff99e07d7\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " Oct 30 13:39:00.247095 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.246978 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c666cfb4-8b66-49b3-b248-2f2771088444-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c666cfb4-8b66-49b3-b248-2f2771088444-credential-socket\") pod \"c666cfb4-8b66-49b3-b248-2f2771088444\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " Oct 30 13:39:00.247095 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.247019 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c666cfb4-8b66-49b3-b248-2f2771088444-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c666cfb4-8b66-49b3-b248-2f2771088444-workload-certs\") pod \"c666cfb4-8b66-49b3-b248-2f2771088444\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " Oct 30 13:39:00.247095 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.247059 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c666cfb4-8b66-49b3-b248-2f2771088444-kube-api-access-gnrnd\" (UniqueName: \"kubernetes.io/projected/c666cfb4-8b66-49b3-b248-2f2771088444-kube-api-access-gnrnd\") pod \"c666cfb4-8b66-49b3-b248-2f2771088444\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " Oct 30 13:39:00.247327 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.247120 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c666cfb4-8b66-49b3-b248-2f2771088444-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c666cfb4-8b66-49b3-b248-2f2771088444-istiod-ca-cert\") pod \"c666cfb4-8b66-49b3-b248-2f2771088444\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " Oct 30 13:39:00.247327 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.247154 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c666cfb4-8b66-49b3-b248-2f2771088444-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c666cfb4-8b66-49b3-b248-2f2771088444-istio-envoy\") pod \"c666cfb4-8b66-49b3-b248-2f2771088444\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " Oct 30 13:39:00.247327 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.247196 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a38c538-ca9f-4135-96f8-32dff99e07d7-istio-token\" (UniqueName: \"kubernetes.io/projected/6a38c538-ca9f-4135-96f8-32dff99e07d7-istio-token\") pod \"6a38c538-ca9f-4135-96f8-32dff99e07d7\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " Oct 30 13:39:00.247327 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.247300 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a38c538-ca9f-4135-96f8-32dff99e07d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a38c538-ca9f-4135-96f8-32dff99e07d7-istiod-ca-cert\") pod \"6a38c538-ca9f-4135-96f8-32dff99e07d7\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " Oct 30 13:39:00.247570 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.247360 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c666cfb4-8b66-49b3-b248-2f2771088444-istio-token\" (UniqueName: \"kubernetes.io/projected/c666cfb4-8b66-49b3-b248-2f2771088444-istio-token\") pod \"c666cfb4-8b66-49b3-b248-2f2771088444\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " Oct 30 13:39:00.247570 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.247409 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c666cfb4-8b66-49b3-b248-2f2771088444-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c666cfb4-8b66-49b3-b248-2f2771088444-istio-podinfo\") pod \"c666cfb4-8b66-49b3-b248-2f2771088444\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " Oct 30 13:39:00.247570 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.247459 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a38c538-ca9f-4135-96f8-32dff99e07d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a38c538-ca9f-4135-96f8-32dff99e07d7-workload-socket\") pod \"6a38c538-ca9f-4135-96f8-32dff99e07d7\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " Oct 30 13:39:00.247736 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.247603 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a38c538-ca9f-4135-96f8-32dff99e07d7-kube-api-access-dc46j\" (UniqueName: \"kubernetes.io/projected/6a38c538-ca9f-4135-96f8-32dff99e07d7-kube-api-access-dc46j\") pod \"6a38c538-ca9f-4135-96f8-32dff99e07d7\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " Oct 30 13:39:00.247736 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.247633 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a38c538-ca9f-4135-96f8-32dff99e07d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a38c538-ca9f-4135-96f8-32dff99e07d7-credential-socket\") pod \"6a38c538-ca9f-4135-96f8-32dff99e07d7\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " Oct 30 13:39:00.247736 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.247659 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c666cfb4-8b66-49b3-b248-2f2771088444-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c666cfb4-8b66-49b3-b248-2f2771088444-workload-socket\") pod \"c666cfb4-8b66-49b3-b248-2f2771088444\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " Oct 30 13:39:00.247736 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.247710 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a38c538-ca9f-4135-96f8-32dff99e07d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a38c538-ca9f-4135-96f8-32dff99e07d7-workload-certs\") pod \"6a38c538-ca9f-4135-96f8-32dff99e07d7\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " Oct 30 13:39:00.249598 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.248332 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a38c538-ca9f-4135-96f8-32dff99e07d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a38c538-ca9f-4135-96f8-32dff99e07d7-istio-envoy\") pod \"6a38c538-ca9f-4135-96f8-32dff99e07d7\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " Oct 30 13:39:00.249598 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.248409 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a38c538-ca9f-4135-96f8-32dff99e07d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a38c538-ca9f-4135-96f8-32dff99e07d7-istio-data\") pod \"6a38c538-ca9f-4135-96f8-32dff99e07d7\" (UID: \"6a38c538-ca9f-4135-96f8-32dff99e07d7\") " Oct 30 13:39:00.249598 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.248460 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c666cfb4-8b66-49b3-b248-2f2771088444-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c666cfb4-8b66-49b3-b248-2f2771088444-istio-data\") pod \"c666cfb4-8b66-49b3-b248-2f2771088444\" (UID: \"c666cfb4-8b66-49b3-b248-2f2771088444\") " Oct 30 13:39:00.249598 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.249485 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c666cfb4-8b66-49b3-b248-2f2771088444-istio-data" pod "c666cfb4-8b66-49b3-b248-2f2771088444" (UID: "c666cfb4-8b66-49b3-b248-2f2771088444"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:00.250625 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.250371 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a38c538-ca9f-4135-96f8-32dff99e07d7-credential-socket" pod "6a38c538-ca9f-4135-96f8-32dff99e07d7" (UID: "6a38c538-ca9f-4135-96f8-32dff99e07d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:00.251115 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.250966 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a38c538-ca9f-4135-96f8-32dff99e07d7-workload-certs" pod "6a38c538-ca9f-4135-96f8-32dff99e07d7" (UID: "6a38c538-ca9f-4135-96f8-32dff99e07d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:00.251552 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.251303 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c666cfb4-8b66-49b3-b248-2f2771088444-workload-socket" pod "c666cfb4-8b66-49b3-b248-2f2771088444" (UID: "c666cfb4-8b66-49b3-b248-2f2771088444"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:00.253060 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.252867 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a38c538-ca9f-4135-96f8-32dff99e07d7-istio-podinfo" pod "6a38c538-ca9f-4135-96f8-32dff99e07d7" (UID: "6a38c538-ca9f-4135-96f8-32dff99e07d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:39:00.253363 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.253180 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a38c538-ca9f-4135-96f8-32dff99e07d7-istio-data" pod "6a38c538-ca9f-4135-96f8-32dff99e07d7" (UID: "6a38c538-ca9f-4135-96f8-32dff99e07d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:00.253808 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.253596 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c666cfb4-8b66-49b3-b248-2f2771088444-istiod-ca-cert" pod "c666cfb4-8b66-49b3-b248-2f2771088444" (UID: "c666cfb4-8b66-49b3-b248-2f2771088444"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:39:00.255312 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.255131 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c666cfb4-8b66-49b3-b248-2f2771088444-istio-token" pod "c666cfb4-8b66-49b3-b248-2f2771088444" (UID: "c666cfb4-8b66-49b3-b248-2f2771088444"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:00.255417 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.255345 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c666cfb4-8b66-49b3-b248-2f2771088444-workload-certs" pod "c666cfb4-8b66-49b3-b248-2f2771088444" (UID: "c666cfb4-8b66-49b3-b248-2f2771088444"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:00.255480 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.255448 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c666cfb4-8b66-49b3-b248-2f2771088444-credential-socket" pod "c666cfb4-8b66-49b3-b248-2f2771088444" (UID: "c666cfb4-8b66-49b3-b248-2f2771088444"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:00.255610 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.255578 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a38c538-ca9f-4135-96f8-32dff99e07d7-workload-socket" pod "6a38c538-ca9f-4135-96f8-32dff99e07d7" (UID: "6a38c538-ca9f-4135-96f8-32dff99e07d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:00.255903 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.255880 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a38c538-ca9f-4135-96f8-32dff99e07d7-istiod-ca-cert" pod "6a38c538-ca9f-4135-96f8-32dff99e07d7" (UID: "6a38c538-ca9f-4135-96f8-32dff99e07d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:39:00.256588 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.256361 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a38c538-ca9f-4135-96f8-32dff99e07d7-istio-token" pod "6a38c538-ca9f-4135-96f8-32dff99e07d7" (UID: "6a38c538-ca9f-4135-96f8-32dff99e07d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:00.256782 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.256748 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a38c538-ca9f-4135-96f8-32dff99e07d7-istio-envoy" pod "6a38c538-ca9f-4135-96f8-32dff99e07d7" (UID: "6a38c538-ca9f-4135-96f8-32dff99e07d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:00.257965 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.257942 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c666cfb4-8b66-49b3-b248-2f2771088444-istio-podinfo" pod "c666cfb4-8b66-49b3-b248-2f2771088444" (UID: "c666cfb4-8b66-49b3-b248-2f2771088444"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:39:00.258064 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.257988 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c666cfb4-8b66-49b3-b248-2f2771088444-kube-api-access-gnrnd" pod "c666cfb4-8b66-49b3-b248-2f2771088444" (UID: "c666cfb4-8b66-49b3-b248-2f2771088444"). InnerVolumeSpecName "kube-api-access-gnrnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:00.259181 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.259154 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c666cfb4-8b66-49b3-b248-2f2771088444-istio-envoy" pod "c666cfb4-8b66-49b3-b248-2f2771088444" (UID: "c666cfb4-8b66-49b3-b248-2f2771088444"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:00.259599 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.259578 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a38c538-ca9f-4135-96f8-32dff99e07d7-kube-api-access-dc46j" pod "6a38c538-ca9f-4135-96f8-32dff99e07d7" (UID: "6a38c538-ca9f-4135-96f8-32dff99e07d7"). InnerVolumeSpecName "kube-api-access-dc46j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:00.307461 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.307431 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" Oct 30 13:39:00.350368 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.350337 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c6dbc7e-8785-4971-88c4-e3341faaadff-kube-api-access-kpz7s\" (UniqueName: \"kubernetes.io/projected/5c6dbc7e-8785-4971-88c4-e3341faaadff-kube-api-access-kpz7s\") pod \"5c6dbc7e-8785-4971-88c4-e3341faaadff\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " Oct 30 13:39:00.350549 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.350383 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c6dbc7e-8785-4971-88c4-e3341faaadff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c6dbc7e-8785-4971-88c4-e3341faaadff-istio-envoy\") pod \"5c6dbc7e-8785-4971-88c4-e3341faaadff\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " Oct 30 13:39:00.350549 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.350417 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c6dbc7e-8785-4971-88c4-e3341faaadff-istio-token\" (UniqueName: \"kubernetes.io/projected/5c6dbc7e-8785-4971-88c4-e3341faaadff-istio-token\") pod \"5c6dbc7e-8785-4971-88c4-e3341faaadff\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " Oct 30 13:39:00.350549 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.350497 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c6dbc7e-8785-4971-88c4-e3341faaadff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6dbc7e-8785-4971-88c4-e3341faaadff-workload-socket\") pod \"5c6dbc7e-8785-4971-88c4-e3341faaadff\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " Oct 30 13:39:00.350549 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.350544 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c6dbc7e-8785-4971-88c4-e3341faaadff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6dbc7e-8785-4971-88c4-e3341faaadff-credential-socket\") pod \"5c6dbc7e-8785-4971-88c4-e3341faaadff\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " Oct 30 13:39:00.350787 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.350602 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c6dbc7e-8785-4971-88c4-e3341faaadff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c6dbc7e-8785-4971-88c4-e3341faaadff-istiod-ca-cert\") pod \"5c6dbc7e-8785-4971-88c4-e3341faaadff\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " Oct 30 13:39:00.350787 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.350668 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c6dbc7e-8785-4971-88c4-e3341faaadff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c6dbc7e-8785-4971-88c4-e3341faaadff-workload-certs\") pod \"5c6dbc7e-8785-4971-88c4-e3341faaadff\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " Oct 30 13:39:00.350787 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.350732 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c6dbc7e-8785-4971-88c4-e3341faaadff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c6dbc7e-8785-4971-88c4-e3341faaadff-istio-data\") pod \"5c6dbc7e-8785-4971-88c4-e3341faaadff\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " Oct 30 13:39:00.350787 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.350782 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c6dbc7e-8785-4971-88c4-e3341faaadff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c6dbc7e-8785-4971-88c4-e3341faaadff-istio-podinfo\") pod \"5c6dbc7e-8785-4971-88c4-e3341faaadff\" (UID: \"5c6dbc7e-8785-4971-88c4-e3341faaadff\") " Oct 30 13:39:00.351022 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.350824 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6dbc7e-8785-4971-88c4-e3341faaadff-workload-socket" pod "5c6dbc7e-8785-4971-88c4-e3341faaadff" (UID: "5c6dbc7e-8785-4971-88c4-e3341faaadff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:00.351022 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.350857 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6dbc7e-8785-4971-88c4-e3341faaadff-credential-socket" pod "5c6dbc7e-8785-4971-88c4-e3341faaadff" (UID: "5c6dbc7e-8785-4971-88c4-e3341faaadff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:00.351022 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.350973 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c6dbc7e-8785-4971-88c4-e3341faaadff-istiod-ca-cert" pod "5c6dbc7e-8785-4971-88c4-e3341faaadff" (UID: "5c6dbc7e-8785-4971-88c4-e3341faaadff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:39:00.351170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.351077 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6dbc7e-8785-4971-88c4-e3341faaadff-istio-data" pod "5c6dbc7e-8785-4971-88c4-e3341faaadff" (UID: "5c6dbc7e-8785-4971-88c4-e3341faaadff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:00.351170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.351090 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6dbc7e-8785-4971-88c4-e3341faaadff-workload-certs" pod "5c6dbc7e-8785-4971-88c4-e3341faaadff" (UID: "5c6dbc7e-8785-4971-88c4-e3341faaadff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:00.351648 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.351623 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c666cfb4-8b66-49b3-b248-2f2771088444-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:00.351703 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.351657 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c666cfb4-8b66-49b3-b248-2f2771088444-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:00.351703 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.351673 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a38c538-ca9f-4135-96f8-32dff99e07d7-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:00.351703 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.351687 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a38c538-ca9f-4135-96f8-32dff99e07d7-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:00.351703 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.351700 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c666cfb4-8b66-49b3-b248-2f2771088444-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:00.351840 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.351714 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c666cfb4-8b66-49b3-b248-2f2771088444-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:00.351840 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.351726 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a38c538-ca9f-4135-96f8-32dff99e07d7-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:00.351840 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.351743 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc46j\" (UniqueName: \"kubernetes.io/projected/6a38c538-ca9f-4135-96f8-32dff99e07d7-kube-api-access-dc46j\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:00.351840 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.351756 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a38c538-ca9f-4135-96f8-32dff99e07d7-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:00.351840 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.351772 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c666cfb4-8b66-49b3-b248-2f2771088444-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:00.351840 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.351785 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a38c538-ca9f-4135-96f8-32dff99e07d7-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:00.351840 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.351799 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6dbc7e-8785-4971-88c4-e3341faaadff-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:00.351840 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.351812 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a38c538-ca9f-4135-96f8-32dff99e07d7-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:00.351840 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.351825 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a38c538-ca9f-4135-96f8-32dff99e07d7-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:00.351840 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.351840 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6dbc7e-8785-4971-88c4-e3341faaadff-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:00.352161 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.351855 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c666cfb4-8b66-49b3-b248-2f2771088444-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:00.352161 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.351867 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a38c538-ca9f-4135-96f8-32dff99e07d7-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:00.352161 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.351880 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c6dbc7e-8785-4971-88c4-e3341faaadff-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:00.352161 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.351894 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c666cfb4-8b66-49b3-b248-2f2771088444-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:00.352161 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.351908 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c6dbc7e-8785-4971-88c4-e3341faaadff-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:00.352161 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.351922 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c666cfb4-8b66-49b3-b248-2f2771088444-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:00.352161 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.351936 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnrnd\" (UniqueName: \"kubernetes.io/projected/c666cfb4-8b66-49b3-b248-2f2771088444-kube-api-access-gnrnd\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:00.352161 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.351948 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c6dbc7e-8785-4971-88c4-e3341faaadff-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:00.352747 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.352721 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6dbc7e-8785-4971-88c4-e3341faaadff-istio-envoy" pod "5c6dbc7e-8785-4971-88c4-e3341faaadff" (UID: "5c6dbc7e-8785-4971-88c4-e3341faaadff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:00.352889 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.352870 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c6dbc7e-8785-4971-88c4-e3341faaadff-istio-podinfo" pod "5c6dbc7e-8785-4971-88c4-e3341faaadff" (UID: "5c6dbc7e-8785-4971-88c4-e3341faaadff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:39:00.353172 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.353150 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c6dbc7e-8785-4971-88c4-e3341faaadff-istio-token" pod "5c6dbc7e-8785-4971-88c4-e3341faaadff" (UID: "5c6dbc7e-8785-4971-88c4-e3341faaadff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:00.353871 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.353852 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c6dbc7e-8785-4971-88c4-e3341faaadff-kube-api-access-kpz7s" pod "5c6dbc7e-8785-4971-88c4-e3341faaadff" (UID: "5c6dbc7e-8785-4971-88c4-e3341faaadff"). InnerVolumeSpecName "kube-api-access-kpz7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:00.453027 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.452993 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpz7s\" (UniqueName: \"kubernetes.io/projected/5c6dbc7e-8785-4971-88c4-e3341faaadff-kube-api-access-kpz7s\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:00.453027 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.453026 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c6dbc7e-8785-4971-88c4-e3341faaadff-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:00.453201 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.453040 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c6dbc7e-8785-4971-88c4-e3341faaadff-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:00.453201 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.453053 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c6dbc7e-8785-4971-88c4-e3341faaadff-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:00.767776 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.767741 4931 generic.go:356] "Generic (PLEG): container finished" podID="6a38c538-ca9f-4135-96f8-32dff99e07d7" containerID="094e0849ae68f5e03c375b4456a2242828b445abd1bc0e6d277926a5dcce6e3c" exitCode=0 Oct 30 13:39:00.767979 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.767786 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" Oct 30 13:39:00.767979 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.767823 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" event={"ID":"6a38c538-ca9f-4135-96f8-32dff99e07d7","Type":"ContainerDied","Data":"094e0849ae68f5e03c375b4456a2242828b445abd1bc0e6d277926a5dcce6e3c"} Oct 30 13:39:00.767979 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.767890 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd" event={"ID":"6a38c538-ca9f-4135-96f8-32dff99e07d7","Type":"ContainerDied","Data":"f0e2a66d0199fe2985a7edf099f6b8aa8564ab278ca4c7890331fee6ca688c6e"} Oct 30 13:39:00.767979 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.767926 4931 scope.go:122] "RemoveContainer" containerID="094e0849ae68f5e03c375b4456a2242828b445abd1bc0e6d277926a5dcce6e3c" Oct 30 13:39:00.777828 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.777769 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-acd125-w5qhp" event={"ID":"89b2ce06-3adf-4653-b527-2b1bf252cde6","Type":"ContainerDied","Data":"f0bd45471551fb218c523d69602313f1855aa042683004738bc39ee8ccf171e9"} Oct 30 13:39:00.777983 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.777953 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-acd125-w5qhp" Oct 30 13:39:00.781961 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.781935 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0bd45471551fb218c523d69602313f1855aa042683004738bc39ee8ccf171e9" Oct 30 13:39:00.782110 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.781966 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2afe58-g92hc" event={"ID":"6ba37aa6-09b2-4e31-a7f4-86db3413c202","Type":"ContainerStarted","Data":"e0509e26a8742eef5725f4c7f70b0f9ee0943003fdb2ef25d15209ff4675db67"} Oct 30 13:39:00.782110 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.782010 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2afe58-g92hc" event={"ID":"6ba37aa6-09b2-4e31-a7f4-86db3413c202","Type":"ContainerStarted","Data":"882fb94a6ddaf2cca9a93e53ff3416e3500998e47998c88563a6543d3dd8a6fb"} Oct 30 13:39:00.783879 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.783585 4931 generic.go:356] "Generic (PLEG): container finished" podID="c666cfb4-8b66-49b3-b248-2f2771088444" containerID="b1fbfd0a3711a27de8b03930069de51ec828290cf14a6961383d360484d02453" exitCode=0 Oct 30 13:39:00.783879 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.783713 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" Oct 30 13:39:00.784223 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.784195 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" event={"ID":"c666cfb4-8b66-49b3-b248-2f2771088444","Type":"ContainerDied","Data":"b1fbfd0a3711a27de8b03930069de51ec828290cf14a6961383d360484d02453"} Oct 30 13:39:00.784411 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.784386 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj" event={"ID":"c666cfb4-8b66-49b3-b248-2f2771088444","Type":"ContainerDied","Data":"dcc4dec218149db23ae3e0bf6b58e453c15bc65dbe35bd29459bb40a566e5db6"} Oct 30 13:39:00.786575 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.786550 4931 generic.go:356] "Generic (PLEG): container finished" podID="5c6dbc7e-8785-4971-88c4-e3341faaadff" containerID="9148f6cf302e45329dca322c30a067837c39207a4d11f48b7cb56acb3493f713" exitCode=0 Oct 30 13:39:00.786736 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.786702 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" event={"ID":"5c6dbc7e-8785-4971-88c4-e3341faaadff","Type":"ContainerDied","Data":"9148f6cf302e45329dca322c30a067837c39207a4d11f48b7cb56acb3493f713"} Oct 30 13:39:00.786828 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.786741 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" event={"ID":"5c6dbc7e-8785-4971-88c4-e3341faaadff","Type":"ContainerDied","Data":"7cd31c858742de0261dbdd8b7b2cee86600e83220a5fa3b474c154d202aa475a"} Oct 30 13:39:00.786828 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.786811 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg" Oct 30 13:39:00.797007 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.796989 4931 scope.go:122] "RemoveContainer" containerID="094e0849ae68f5e03c375b4456a2242828b445abd1bc0e6d277926a5dcce6e3c" Oct 30 13:39:00.797292 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:39:00.797258 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"094e0849ae68f5e03c375b4456a2242828b445abd1bc0e6d277926a5dcce6e3c\": container with ID starting with 094e0849ae68f5e03c375b4456a2242828b445abd1bc0e6d277926a5dcce6e3c not found: ID does not exist" containerID="094e0849ae68f5e03c375b4456a2242828b445abd1bc0e6d277926a5dcce6e3c" Oct 30 13:39:00.797391 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.797298 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"094e0849ae68f5e03c375b4456a2242828b445abd1bc0e6d277926a5dcce6e3c"} err="failed to get container status \"094e0849ae68f5e03c375b4456a2242828b445abd1bc0e6d277926a5dcce6e3c\": rpc error: code = NotFound desc = could not find container \"094e0849ae68f5e03c375b4456a2242828b445abd1bc0e6d277926a5dcce6e3c\": container with ID starting with 094e0849ae68f5e03c375b4456a2242828b445abd1bc0e6d277926a5dcce6e3c not found: ID does not exist" Oct 30 13:39:00.797391 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.797314 4931 scope.go:122] "RemoveContainer" containerID="b1fbfd0a3711a27de8b03930069de51ec828290cf14a6961383d360484d02453" Oct 30 13:39:00.808056 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.808036 4931 scope.go:122] "RemoveContainer" containerID="b1fbfd0a3711a27de8b03930069de51ec828290cf14a6961383d360484d02453" Oct 30 13:39:00.808381 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:39:00.808359 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1fbfd0a3711a27de8b03930069de51ec828290cf14a6961383d360484d02453\": container with ID starting with b1fbfd0a3711a27de8b03930069de51ec828290cf14a6961383d360484d02453 not found: ID does not exist" containerID="b1fbfd0a3711a27de8b03930069de51ec828290cf14a6961383d360484d02453" Oct 30 13:39:00.808438 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.808390 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1fbfd0a3711a27de8b03930069de51ec828290cf14a6961383d360484d02453"} err="failed to get container status \"b1fbfd0a3711a27de8b03930069de51ec828290cf14a6961383d360484d02453\": rpc error: code = NotFound desc = could not find container \"b1fbfd0a3711a27de8b03930069de51ec828290cf14a6961383d360484d02453\": container with ID starting with b1fbfd0a3711a27de8b03930069de51ec828290cf14a6961383d360484d02453 not found: ID does not exist" Oct 30 13:39:00.808438 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.808410 4931 scope.go:122] "RemoveContainer" containerID="9148f6cf302e45329dca322c30a067837c39207a4d11f48b7cb56acb3493f713" Oct 30 13:39:00.819156 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.819093 4931 scope.go:122] "RemoveContainer" containerID="9148f6cf302e45329dca322c30a067837c39207a4d11f48b7cb56acb3493f713" Oct 30 13:39:00.819557 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:39:00.819535 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9148f6cf302e45329dca322c30a067837c39207a4d11f48b7cb56acb3493f713\": container with ID starting with 9148f6cf302e45329dca322c30a067837c39207a4d11f48b7cb56acb3493f713 not found: ID does not exist" containerID="9148f6cf302e45329dca322c30a067837c39207a4d11f48b7cb56acb3493f713" Oct 30 13:39:00.819643 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.819565 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9148f6cf302e45329dca322c30a067837c39207a4d11f48b7cb56acb3493f713"} err="failed to get container status \"9148f6cf302e45329dca322c30a067837c39207a4d11f48b7cb56acb3493f713\": rpc error: code = NotFound desc = could not find container \"9148f6cf302e45329dca322c30a067837c39207a4d11f48b7cb56acb3493f713\": container with ID starting with 9148f6cf302e45329dca322c30a067837c39207a4d11f48b7cb56acb3493f713 not found: ID does not exist" Oct 30 13:39:00.829121 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.829077 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2afe58-g92hc" podStartSLOduration=1.829065398 podStartE2EDuration="1.829065398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:38:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:39:00.826876154 +0000 UTC m=+3239.430374599" watchObservedRunningTime="2026-10-30 13:39:00.829065398 +0000 UTC m=+3239.432563845" Oct 30 13:39:00.885063 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.885038 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd"] Oct 30 13:39:00.897445 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.897412 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd"] Oct 30 13:39:00.945039 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.945008 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj"] Oct 30 13:39:00.949035 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.949005 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj"] Oct 30 13:39:00.992206 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:00.992174 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg"] Oct 30 13:39:01.000689 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:01.000660 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg"] Oct 30 13:39:01.514901 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:01.514828 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c6dbc7e-8785-4971-88c4-e3341faaadff" path="/var/lib/kubelet/pods/5c6dbc7e-8785-4971-88c4-e3341faaadff/volumes" Oct 30 13:39:01.515396 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:01.515380 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a38c538-ca9f-4135-96f8-32dff99e07d7" path="/var/lib/kubelet/pods/6a38c538-ca9f-4135-96f8-32dff99e07d7/volumes" Oct 30 13:39:01.515867 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:01.515854 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c666cfb4-8b66-49b3-b248-2f2771088444" path="/var/lib/kubelet/pods/c666cfb4-8b66-49b3-b248-2f2771088444/volumes" Oct 30 13:39:01.798806 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:01.798723 4931 generic.go:356] "Generic (PLEG): container finished" podID="6ba37aa6-09b2-4e31-a7f4-86db3413c202" containerID="e0509e26a8742eef5725f4c7f70b0f9ee0943003fdb2ef25d15209ff4675db67" exitCode=0 Oct 30 13:39:01.798973 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:01.798809 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2afe58-g92hc" event={"ID":"6ba37aa6-09b2-4e31-a7f4-86db3413c202","Type":"ContainerDied","Data":"e0509e26a8742eef5725f4c7f70b0f9ee0943003fdb2ef25d15209ff4675db67"} Oct 30 13:39:02.494781 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:02.494751 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft"] Oct 30 13:39:02.495280 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:02.494956 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" podUID="83287de4-d8f9-4a84-96df-be7522695d0b" containerName="istio-proxy" containerID="cri-o://a92616be2650f4c91ab2936612328448d7fc2ec46e0128967022d58879920352" gracePeriod=30 Oct 30 13:39:02.979038 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:02.979008 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2afe58-g92hc" Oct 30 13:39:03.109791 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:03.109700 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ba37aa6-09b2-4e31-a7f4-86db3413c202-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ba37aa6-09b2-4e31-a7f4-86db3413c202-maas-api-openshift-ca\") pod \"6ba37aa6-09b2-4e31-a7f4-86db3413c202\" (UID: \"6ba37aa6-09b2-4e31-a7f4-86db3413c202\") " Oct 30 13:39:03.109995 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:03.109857 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ba37aa6-09b2-4e31-a7f4-86db3413c202-kube-api-access-jv5pd\" (UniqueName: \"kubernetes.io/projected/6ba37aa6-09b2-4e31-a7f4-86db3413c202-kube-api-access-jv5pd\") pod \"6ba37aa6-09b2-4e31-a7f4-86db3413c202\" (UID: \"6ba37aa6-09b2-4e31-a7f4-86db3413c202\") " Oct 30 13:39:03.109995 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:03.109938 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ba37aa6-09b2-4e31-a7f4-86db3413c202-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ba37aa6-09b2-4e31-a7f4-86db3413c202-maas-api-xks-ca\") pod \"6ba37aa6-09b2-4e31-a7f4-86db3413c202\" (UID: \"6ba37aa6-09b2-4e31-a7f4-86db3413c202\") " Oct 30 13:39:03.110133 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:03.110055 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ba37aa6-09b2-4e31-a7f4-86db3413c202-maas-api-openshift-ca" pod "6ba37aa6-09b2-4e31-a7f4-86db3413c202" (UID: "6ba37aa6-09b2-4e31-a7f4-86db3413c202"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:39:03.110874 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:03.110844 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ba37aa6-09b2-4e31-a7f4-86db3413c202-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:03.112102 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:03.112076 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ba37aa6-09b2-4e31-a7f4-86db3413c202-maas-api-xks-ca" pod "6ba37aa6-09b2-4e31-a7f4-86db3413c202" (UID: "6ba37aa6-09b2-4e31-a7f4-86db3413c202"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:39:03.112565 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:03.112540 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba37aa6-09b2-4e31-a7f4-86db3413c202-kube-api-access-jv5pd" pod "6ba37aa6-09b2-4e31-a7f4-86db3413c202" (UID: "6ba37aa6-09b2-4e31-a7f4-86db3413c202"). InnerVolumeSpecName "kube-api-access-jv5pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:03.212402 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:03.212369 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv5pd\" (UniqueName: \"kubernetes.io/projected/6ba37aa6-09b2-4e31-a7f4-86db3413c202-kube-api-access-jv5pd\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:03.212402 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:03.212399 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ba37aa6-09b2-4e31-a7f4-86db3413c202-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:03.818614 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:03.818526 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2afe58-g92hc" Oct 30 13:39:03.819006 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:03.818531 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2afe58-g92hc" event={"ID":"6ba37aa6-09b2-4e31-a7f4-86db3413c202","Type":"ContainerDied","Data":"882fb94a6ddaf2cca9a93e53ff3416e3500998e47998c88563a6543d3dd8a6fb"} Oct 30 13:39:03.819006 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:03.818706 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="882fb94a6ddaf2cca9a93e53ff3416e3500998e47998c88563a6543d3dd8a6fb" Oct 30 13:39:04.172946 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.172887 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" Oct 30 13:39:04.220453 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.220415 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" Oct 30 13:39:04.229150 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.228825 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-istio-token\" (UniqueName: \"kubernetes.io/projected/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-istio-token\") pod \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " Oct 30 13:39:04.229150 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.229026 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-kube-api-access-d5b5p\" (UniqueName: \"kubernetes.io/projected/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-kube-api-access-d5b5p\") pod \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " Oct 30 13:39:04.229150 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.229073 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-workload-socket\") pod \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " Oct 30 13:39:04.229150 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.229137 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-istio-envoy\") pod \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " Oct 30 13:39:04.229883 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.229218 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-istio-data\") pod \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " Oct 30 13:39:04.229883 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.229244 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-istiod-ca-cert\") pod \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " Oct 30 13:39:04.229883 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.229339 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-istio-podinfo\") pod \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " Oct 30 13:39:04.229883 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.229393 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-credential-socket\") pod \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " Oct 30 13:39:04.229883 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.229434 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-workload-certs\") pod \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\" (UID: \"813ffc2b-3c94-4847-aa9b-2d7971da6e4d\") " Oct 30 13:39:04.229883 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.229556 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-workload-socket" pod "813ffc2b-3c94-4847-aa9b-2d7971da6e4d" (UID: "813ffc2b-3c94-4847-aa9b-2d7971da6e4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:04.229883 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.229867 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-istiod-ca-cert" pod "813ffc2b-3c94-4847-aa9b-2d7971da6e4d" (UID: "813ffc2b-3c94-4847-aa9b-2d7971da6e4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:39:04.233097 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.230176 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-istio-data" pod "813ffc2b-3c94-4847-aa9b-2d7971da6e4d" (UID: "813ffc2b-3c94-4847-aa9b-2d7971da6e4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:04.233097 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.230758 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-workload-certs" pod "813ffc2b-3c94-4847-aa9b-2d7971da6e4d" (UID: "813ffc2b-3c94-4847-aa9b-2d7971da6e4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:04.233097 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.231083 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:04.233097 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.231104 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:04.233097 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.231117 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:04.233097 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.231130 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:04.234927 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.234898 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-istio-token" pod "813ffc2b-3c94-4847-aa9b-2d7971da6e4d" (UID: "813ffc2b-3c94-4847-aa9b-2d7971da6e4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:04.235030 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.234924 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-credential-socket" pod "813ffc2b-3c94-4847-aa9b-2d7971da6e4d" (UID: "813ffc2b-3c94-4847-aa9b-2d7971da6e4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:04.237075 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.237050 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-istio-envoy" pod "813ffc2b-3c94-4847-aa9b-2d7971da6e4d" (UID: "813ffc2b-3c94-4847-aa9b-2d7971da6e4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:04.237700 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.237676 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-kube-api-access-d5b5p" pod "813ffc2b-3c94-4847-aa9b-2d7971da6e4d" (UID: "813ffc2b-3c94-4847-aa9b-2d7971da6e4d"). InnerVolumeSpecName "kube-api-access-d5b5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:04.240417 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.240394 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-istio-podinfo" pod "813ffc2b-3c94-4847-aa9b-2d7971da6e4d" (UID: "813ffc2b-3c94-4847-aa9b-2d7971da6e4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:39:04.262951 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.262930 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" Oct 30 13:39:04.307304 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.307254 4931 scope.go:122] "RemoveContainer" containerID="bff679f4cf90770700d94def65e70afa21b10a539c50e38078591749a8784758" Oct 30 13:39:04.320750 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.320727 4931 scope.go:122] "RemoveContainer" containerID="14fa7bf0d4face1d856677ad9c6ac64211154f7bd1c11e56050677a2c3c873e6" Oct 30 13:39:04.331601 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.331571 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6f71a7e-fbeb-4e16-8863-e17fc9695553-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6f71a7e-fbeb-4e16-8863-e17fc9695553-istio-podinfo\") pod \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " Oct 30 13:39:04.331725 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.331619 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9916ba3d-3bef-4338-aa2e-f5c27782188f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9916ba3d-3bef-4338-aa2e-f5c27782188f-istio-podinfo\") pod \"9916ba3d-3bef-4338-aa2e-f5c27782188f\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " Oct 30 13:39:04.331725 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.331646 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6f71a7e-fbeb-4e16-8863-e17fc9695553-kube-api-access-kl4qt\" (UniqueName: \"kubernetes.io/projected/d6f71a7e-fbeb-4e16-8863-e17fc9695553-kube-api-access-kl4qt\") pod \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " Oct 30 13:39:04.331725 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.331686 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f71a7e-fbeb-4e16-8863-e17fc9695553-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f71a7e-fbeb-4e16-8863-e17fc9695553-workload-socket\") pod \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " Oct 30 13:39:04.331924 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.331777 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f71a7e-fbeb-4e16-8863-e17fc9695553-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6f71a7e-fbeb-4e16-8863-e17fc9695553-workload-certs\") pod \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " Oct 30 13:39:04.331924 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.331808 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f71a7e-fbeb-4e16-8863-e17fc9695553-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6f71a7e-fbeb-4e16-8863-e17fc9695553-istio-data\") pod \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " Oct 30 13:39:04.331924 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.331864 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9916ba3d-3bef-4338-aa2e-f5c27782188f-istio-token\" (UniqueName: \"kubernetes.io/projected/9916ba3d-3bef-4338-aa2e-f5c27782188f-istio-token\") pod \"9916ba3d-3bef-4338-aa2e-f5c27782188f\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " Oct 30 13:39:04.331924 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.331891 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9916ba3d-3bef-4338-aa2e-f5c27782188f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9916ba3d-3bef-4338-aa2e-f5c27782188f-istiod-ca-cert\") pod \"9916ba3d-3bef-4338-aa2e-f5c27782188f\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " Oct 30 13:39:04.332145 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.331949 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9916ba3d-3bef-4338-aa2e-f5c27782188f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9916ba3d-3bef-4338-aa2e-f5c27782188f-workload-socket\") pod \"9916ba3d-3bef-4338-aa2e-f5c27782188f\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " Oct 30 13:39:04.332145 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.331996 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f71a7e-fbeb-4e16-8863-e17fc9695553-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f71a7e-fbeb-4e16-8863-e17fc9695553-credential-socket\") pod \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " Oct 30 13:39:04.332145 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.332052 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9916ba3d-3bef-4338-aa2e-f5c27782188f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9916ba3d-3bef-4338-aa2e-f5c27782188f-istio-data\") pod \"9916ba3d-3bef-4338-aa2e-f5c27782188f\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " Oct 30 13:39:04.332145 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.332068 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9916ba3d-3bef-4338-aa2e-f5c27782188f-kube-api-access-b7mcv\" (UniqueName: \"kubernetes.io/projected/9916ba3d-3bef-4338-aa2e-f5c27782188f-kube-api-access-b7mcv\") pod \"9916ba3d-3bef-4338-aa2e-f5c27782188f\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " Oct 30 13:39:04.332145 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.332106 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9916ba3d-3bef-4338-aa2e-f5c27782188f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9916ba3d-3bef-4338-aa2e-f5c27782188f-workload-certs\") pod \"9916ba3d-3bef-4338-aa2e-f5c27782188f\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " Oct 30 13:39:04.332466 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.332176 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9916ba3d-3bef-4338-aa2e-f5c27782188f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9916ba3d-3bef-4338-aa2e-f5c27782188f-istio-envoy\") pod \"9916ba3d-3bef-4338-aa2e-f5c27782188f\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " Oct 30 13:39:04.332466 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.332196 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9916ba3d-3bef-4338-aa2e-f5c27782188f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9916ba3d-3bef-4338-aa2e-f5c27782188f-credential-socket\") pod \"9916ba3d-3bef-4338-aa2e-f5c27782188f\" (UID: \"9916ba3d-3bef-4338-aa2e-f5c27782188f\") " Oct 30 13:39:04.332466 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.332239 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f71a7e-fbeb-4e16-8863-e17fc9695553-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6f71a7e-fbeb-4e16-8863-e17fc9695553-istio-envoy\") pod \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " Oct 30 13:39:04.332466 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.332345 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6f71a7e-fbeb-4e16-8863-e17fc9695553-istio-token\" (UniqueName: \"kubernetes.io/projected/d6f71a7e-fbeb-4e16-8863-e17fc9695553-istio-token\") pod \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " Oct 30 13:39:04.332466 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.332368 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6f71a7e-fbeb-4e16-8863-e17fc9695553-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6f71a7e-fbeb-4e16-8863-e17fc9695553-istiod-ca-cert\") pod \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\" (UID: \"d6f71a7e-fbeb-4e16-8863-e17fc9695553\") " Oct 30 13:39:04.332961 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.332924 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5b5p\" (UniqueName: \"kubernetes.io/projected/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-kube-api-access-d5b5p\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:04.332961 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.332955 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:04.333152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.332975 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:04.333152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.332989 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:04.333152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.333002 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/813ffc2b-3c94-4847-aa9b-2d7971da6e4d-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:04.333885 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.333508 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f71a7e-fbeb-4e16-8863-e17fc9695553-credential-socket" pod "d6f71a7e-fbeb-4e16-8863-e17fc9695553" (UID: "d6f71a7e-fbeb-4e16-8863-e17fc9695553"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:04.333885 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.333540 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6f71a7e-fbeb-4e16-8863-e17fc9695553-istiod-ca-cert" pod "d6f71a7e-fbeb-4e16-8863-e17fc9695553" (UID: "d6f71a7e-fbeb-4e16-8863-e17fc9695553"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:39:04.333885 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.333827 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9916ba3d-3bef-4338-aa2e-f5c27782188f-istio-data" pod "9916ba3d-3bef-4338-aa2e-f5c27782188f" (UID: "9916ba3d-3bef-4338-aa2e-f5c27782188f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:04.334367 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.334343 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f71a7e-fbeb-4e16-8863-e17fc9695553-istio-data" pod "d6f71a7e-fbeb-4e16-8863-e17fc9695553" (UID: "d6f71a7e-fbeb-4e16-8863-e17fc9695553"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:04.334496 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.334471 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f71a7e-fbeb-4e16-8863-e17fc9695553-workload-socket" pod "d6f71a7e-fbeb-4e16-8863-e17fc9695553" (UID: "d6f71a7e-fbeb-4e16-8863-e17fc9695553"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:04.334922 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.334898 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9916ba3d-3bef-4338-aa2e-f5c27782188f-workload-socket" pod "9916ba3d-3bef-4338-aa2e-f5c27782188f" (UID: "9916ba3d-3bef-4338-aa2e-f5c27782188f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:04.335048 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.334963 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9916ba3d-3bef-4338-aa2e-f5c27782188f-istiod-ca-cert" pod "9916ba3d-3bef-4338-aa2e-f5c27782188f" (UID: "9916ba3d-3bef-4338-aa2e-f5c27782188f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:39:04.335048 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.334995 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f71a7e-fbeb-4e16-8863-e17fc9695553-workload-certs" pod "d6f71a7e-fbeb-4e16-8863-e17fc9695553" (UID: "d6f71a7e-fbeb-4e16-8863-e17fc9695553"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:04.335048 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.334971 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9916ba3d-3bef-4338-aa2e-f5c27782188f-istio-podinfo" pod "9916ba3d-3bef-4338-aa2e-f5c27782188f" (UID: "9916ba3d-3bef-4338-aa2e-f5c27782188f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:39:04.335246 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.335221 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9916ba3d-3bef-4338-aa2e-f5c27782188f-credential-socket" pod "9916ba3d-3bef-4338-aa2e-f5c27782188f" (UID: "9916ba3d-3bef-4338-aa2e-f5c27782188f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:04.335550 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.335522 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9916ba3d-3bef-4338-aa2e-f5c27782188f-workload-certs" pod "9916ba3d-3bef-4338-aa2e-f5c27782188f" (UID: "9916ba3d-3bef-4338-aa2e-f5c27782188f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:04.336559 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.336062 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6f71a7e-fbeb-4e16-8863-e17fc9695553-istio-podinfo" pod "d6f71a7e-fbeb-4e16-8863-e17fc9695553" (UID: "d6f71a7e-fbeb-4e16-8863-e17fc9695553"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:39:04.336971 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.336677 4931 scope.go:122] "RemoveContainer" containerID="5817c5f03afa1158a786dbd430b3b2f9b8269ea4bc5573f6fc18e37ca7df626f" Oct 30 13:39:04.341207 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.339415 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9916ba3d-3bef-4338-aa2e-f5c27782188f-kube-api-access-b7mcv" pod "9916ba3d-3bef-4338-aa2e-f5c27782188f" (UID: "9916ba3d-3bef-4338-aa2e-f5c27782188f"). InnerVolumeSpecName "kube-api-access-b7mcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:04.341207 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.339707 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9916ba3d-3bef-4338-aa2e-f5c27782188f-istio-envoy" pod "9916ba3d-3bef-4338-aa2e-f5c27782188f" (UID: "9916ba3d-3bef-4338-aa2e-f5c27782188f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:04.341207 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.340137 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9916ba3d-3bef-4338-aa2e-f5c27782188f-istio-token" pod "9916ba3d-3bef-4338-aa2e-f5c27782188f" (UID: "9916ba3d-3bef-4338-aa2e-f5c27782188f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:04.345647 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.345626 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6f71a7e-fbeb-4e16-8863-e17fc9695553-kube-api-access-kl4qt" pod "d6f71a7e-fbeb-4e16-8863-e17fc9695553" (UID: "d6f71a7e-fbeb-4e16-8863-e17fc9695553"). InnerVolumeSpecName "kube-api-access-kl4qt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:04.347056 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.347031 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6f71a7e-fbeb-4e16-8863-e17fc9695553-istio-token" pod "d6f71a7e-fbeb-4e16-8863-e17fc9695553" (UID: "d6f71a7e-fbeb-4e16-8863-e17fc9695553"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:04.347359 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.347332 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f71a7e-fbeb-4e16-8863-e17fc9695553-istio-envoy" pod "d6f71a7e-fbeb-4e16-8863-e17fc9695553" (UID: "d6f71a7e-fbeb-4e16-8863-e17fc9695553"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:04.353091 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.353036 4931 scope.go:122] "RemoveContainer" containerID="20d5cf0f619190ff7dc8ed463717b550ae60a828e9fbc10a37887a50704e27af" Oct 30 13:39:04.435145 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.434880 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f71a7e-fbeb-4e16-8863-e17fc9695553-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:04.435145 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.434914 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9916ba3d-3bef-4338-aa2e-f5c27782188f-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:04.435145 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.434930 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7mcv\" (UniqueName: \"kubernetes.io/projected/9916ba3d-3bef-4338-aa2e-f5c27782188f-kube-api-access-b7mcv\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:04.435145 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.434946 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9916ba3d-3bef-4338-aa2e-f5c27782188f-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:04.435145 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.434959 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9916ba3d-3bef-4338-aa2e-f5c27782188f-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:04.435145 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.434972 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9916ba3d-3bef-4338-aa2e-f5c27782188f-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:04.435145 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.434984 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6f71a7e-fbeb-4e16-8863-e17fc9695553-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:04.435145 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.434999 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6f71a7e-fbeb-4e16-8863-e17fc9695553-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:04.435145 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.435010 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6f71a7e-fbeb-4e16-8863-e17fc9695553-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:04.435145 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.435023 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6f71a7e-fbeb-4e16-8863-e17fc9695553-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:04.435145 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.435035 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9916ba3d-3bef-4338-aa2e-f5c27782188f-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:04.435145 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.435048 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl4qt\" (UniqueName: \"kubernetes.io/projected/d6f71a7e-fbeb-4e16-8863-e17fc9695553-kube-api-access-kl4qt\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:04.435145 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.435063 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f71a7e-fbeb-4e16-8863-e17fc9695553-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:04.435145 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.435076 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6f71a7e-fbeb-4e16-8863-e17fc9695553-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:04.435145 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.435089 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6f71a7e-fbeb-4e16-8863-e17fc9695553-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:04.435145 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.435101 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9916ba3d-3bef-4338-aa2e-f5c27782188f-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:04.435145 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.435113 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9916ba3d-3bef-4338-aa2e-f5c27782188f-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:04.435145 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.435125 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9916ba3d-3bef-4338-aa2e-f5c27782188f-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:04.827082 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.827045 4931 generic.go:356] "Generic (PLEG): container finished" podID="813ffc2b-3c94-4847-aa9b-2d7971da6e4d" containerID="5498caf666415906672c692ac63dd600423c82d515682a426064d18b60c9fcb7" exitCode=0 Oct 30 13:39:04.827591 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.827095 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" Oct 30 13:39:04.827591 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.827116 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" event={"ID":"813ffc2b-3c94-4847-aa9b-2d7971da6e4d","Type":"ContainerDied","Data":"5498caf666415906672c692ac63dd600423c82d515682a426064d18b60c9fcb7"} Oct 30 13:39:04.827591 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.827172 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm" event={"ID":"813ffc2b-3c94-4847-aa9b-2d7971da6e4d","Type":"ContainerDied","Data":"7710b912d480da97e9a2d1990ef54ba55ea540211f73e8c14c64f6f8af02df1b"} Oct 30 13:39:04.827591 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.827199 4931 scope.go:122] "RemoveContainer" containerID="5498caf666415906672c692ac63dd600423c82d515682a426064d18b60c9fcb7" Oct 30 13:39:04.829045 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.829020 4931 generic.go:356] "Generic (PLEG): container finished" podID="d6f71a7e-fbeb-4e16-8863-e17fc9695553" containerID="0ab016e7e6e72b08b3a2959061f4d839b39b1e4db4d5db283b4375e744948a45" exitCode=0 Oct 30 13:39:04.829177 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.829076 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" Oct 30 13:39:04.829177 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.829104 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" event={"ID":"d6f71a7e-fbeb-4e16-8863-e17fc9695553","Type":"ContainerDied","Data":"0ab016e7e6e72b08b3a2959061f4d839b39b1e4db4d5db283b4375e744948a45"} Oct 30 13:39:04.829177 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.829155 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p" event={"ID":"d6f71a7e-fbeb-4e16-8863-e17fc9695553","Type":"ContainerDied","Data":"9c4548d36d2c6c3db8004a50a3443d3f36e33dd15ceef2e3ada0c7fdf211d154"} Oct 30 13:39:04.830960 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.830931 4931 generic.go:356] "Generic (PLEG): container finished" podID="9916ba3d-3bef-4338-aa2e-f5c27782188f" containerID="84b32dc6f16656f85128e413f1bc9741ef177676729f2f0ba6c25a4ff550df82" exitCode=0 Oct 30 13:39:04.831087 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.831024 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" event={"ID":"9916ba3d-3bef-4338-aa2e-f5c27782188f","Type":"ContainerDied","Data":"84b32dc6f16656f85128e413f1bc9741ef177676729f2f0ba6c25a4ff550df82"} Oct 30 13:39:04.831087 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.831064 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" event={"ID":"9916ba3d-3bef-4338-aa2e-f5c27782188f","Type":"ContainerDied","Data":"6960c197e37643a0841ba0f3dcd2572a2eff526e231eb18a20eef476ed494d2b"} Oct 30 13:39:04.831087 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.831072 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z" Oct 30 13:39:04.839052 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.839031 4931 scope.go:122] "RemoveContainer" containerID="5498caf666415906672c692ac63dd600423c82d515682a426064d18b60c9fcb7" Oct 30 13:39:04.839504 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:39:04.839472 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5498caf666415906672c692ac63dd600423c82d515682a426064d18b60c9fcb7\": container with ID starting with 5498caf666415906672c692ac63dd600423c82d515682a426064d18b60c9fcb7 not found: ID does not exist" containerID="5498caf666415906672c692ac63dd600423c82d515682a426064d18b60c9fcb7" Oct 30 13:39:04.839604 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.839514 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5498caf666415906672c692ac63dd600423c82d515682a426064d18b60c9fcb7"} err="failed to get container status \"5498caf666415906672c692ac63dd600423c82d515682a426064d18b60c9fcb7\": rpc error: code = NotFound desc = could not find container \"5498caf666415906672c692ac63dd600423c82d515682a426064d18b60c9fcb7\": container with ID starting with 5498caf666415906672c692ac63dd600423c82d515682a426064d18b60c9fcb7 not found: ID does not exist" Oct 30 13:39:04.839604 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.839538 4931 scope.go:122] "RemoveContainer" containerID="0ab016e7e6e72b08b3a2959061f4d839b39b1e4db4d5db283b4375e744948a45" Oct 30 13:39:04.850535 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.850517 4931 scope.go:122] "RemoveContainer" containerID="0ab016e7e6e72b08b3a2959061f4d839b39b1e4db4d5db283b4375e744948a45" Oct 30 13:39:04.850768 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:39:04.850751 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ab016e7e6e72b08b3a2959061f4d839b39b1e4db4d5db283b4375e744948a45\": container with ID starting with 0ab016e7e6e72b08b3a2959061f4d839b39b1e4db4d5db283b4375e744948a45 not found: ID does not exist" containerID="0ab016e7e6e72b08b3a2959061f4d839b39b1e4db4d5db283b4375e744948a45" Oct 30 13:39:04.850816 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.850773 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ab016e7e6e72b08b3a2959061f4d839b39b1e4db4d5db283b4375e744948a45"} err="failed to get container status \"0ab016e7e6e72b08b3a2959061f4d839b39b1e4db4d5db283b4375e744948a45\": rpc error: code = NotFound desc = could not find container \"0ab016e7e6e72b08b3a2959061f4d839b39b1e4db4d5db283b4375e744948a45\": container with ID starting with 0ab016e7e6e72b08b3a2959061f4d839b39b1e4db4d5db283b4375e744948a45 not found: ID does not exist" Oct 30 13:39:04.850816 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.850788 4931 scope.go:122] "RemoveContainer" containerID="84b32dc6f16656f85128e413f1bc9741ef177676729f2f0ba6c25a4ff550df82" Oct 30 13:39:04.859943 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.859923 4931 scope.go:122] "RemoveContainer" containerID="84b32dc6f16656f85128e413f1bc9741ef177676729f2f0ba6c25a4ff550df82" Oct 30 13:39:04.860207 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:39:04.860190 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84b32dc6f16656f85128e413f1bc9741ef177676729f2f0ba6c25a4ff550df82\": container with ID starting with 84b32dc6f16656f85128e413f1bc9741ef177676729f2f0ba6c25a4ff550df82 not found: ID does not exist" containerID="84b32dc6f16656f85128e413f1bc9741ef177676729f2f0ba6c25a4ff550df82" Oct 30 13:39:04.860251 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.860212 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84b32dc6f16656f85128e413f1bc9741ef177676729f2f0ba6c25a4ff550df82"} err="failed to get container status \"84b32dc6f16656f85128e413f1bc9741ef177676729f2f0ba6c25a4ff550df82\": rpc error: code = NotFound desc = could not find container \"84b32dc6f16656f85128e413f1bc9741ef177676729f2f0ba6c25a4ff550df82\": container with ID starting with 84b32dc6f16656f85128e413f1bc9741ef177676729f2f0ba6c25a4ff550df82 not found: ID does not exist" Oct 30 13:39:04.911220 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.911192 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm"] Oct 30 13:39:04.924528 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.924500 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm"] Oct 30 13:39:04.955476 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.955448 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p"] Oct 30 13:39:04.963422 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:04.963399 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p"] Oct 30 13:39:05.001367 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:05.001343 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z"] Oct 30 13:39:05.008566 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:05.008545 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z"] Oct 30 13:39:05.374347 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:05.374310 4931 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:39:05.374527 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:05.374376 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" podUID="83287de4-d8f9-4a84-96df-be7522695d0b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:39:05.515005 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:05.514974 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="813ffc2b-3c94-4847-aa9b-2d7971da6e4d" path="/var/lib/kubelet/pods/813ffc2b-3c94-4847-aa9b-2d7971da6e4d/volumes" Oct 30 13:39:05.515549 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:05.515532 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9916ba3d-3bef-4338-aa2e-f5c27782188f" path="/var/lib/kubelet/pods/9916ba3d-3bef-4338-aa2e-f5c27782188f/volumes" Oct 30 13:39:05.515989 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:05.515974 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6f71a7e-fbeb-4e16-8863-e17fc9695553" path="/var/lib/kubelet/pods/d6f71a7e-fbeb-4e16-8863-e17fc9695553/volumes" Oct 30 13:39:07.352879 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:39:07.352843 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:39:07.790646 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:07.790618 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" Oct 30 13:39:07.859046 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:07.859015 4931 generic.go:356] "Generic (PLEG): container finished" podID="83287de4-d8f9-4a84-96df-be7522695d0b" containerID="a92616be2650f4c91ab2936612328448d7fc2ec46e0128967022d58879920352" exitCode=0 Oct 30 13:39:07.859226 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:07.859140 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" Oct 30 13:39:07.859305 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:07.859228 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" event={"ID":"83287de4-d8f9-4a84-96df-be7522695d0b","Type":"ContainerDied","Data":"a92616be2650f4c91ab2936612328448d7fc2ec46e0128967022d58879920352"} Oct 30 13:39:07.859305 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:07.859295 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft" event={"ID":"83287de4-d8f9-4a84-96df-be7522695d0b","Type":"ContainerDied","Data":"1559cf001b0f80d10256eb872d1ea6ce74681834ffc34fae45138450d3879c69"} Oct 30 13:39:07.859442 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:07.859330 4931 scope.go:122] "RemoveContainer" containerID="a92616be2650f4c91ab2936612328448d7fc2ec46e0128967022d58879920352" Oct 30 13:39:07.871845 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:07.871816 4931 scope.go:122] "RemoveContainer" containerID="a92616be2650f4c91ab2936612328448d7fc2ec46e0128967022d58879920352" Oct 30 13:39:07.872159 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:39:07.872133 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a92616be2650f4c91ab2936612328448d7fc2ec46e0128967022d58879920352\": container with ID starting with a92616be2650f4c91ab2936612328448d7fc2ec46e0128967022d58879920352 not found: ID does not exist" containerID="a92616be2650f4c91ab2936612328448d7fc2ec46e0128967022d58879920352" Oct 30 13:39:07.872219 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:07.872172 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a92616be2650f4c91ab2936612328448d7fc2ec46e0128967022d58879920352"} err="failed to get container status \"a92616be2650f4c91ab2936612328448d7fc2ec46e0128967022d58879920352\": rpc error: code = NotFound desc = could not find container \"a92616be2650f4c91ab2936612328448d7fc2ec46e0128967022d58879920352\": container with ID starting with a92616be2650f4c91ab2936612328448d7fc2ec46e0128967022d58879920352 not found: ID does not exist" Oct 30 13:39:07.916669 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:07.916598 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83287de4-d8f9-4a84-96df-be7522695d0b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83287de4-d8f9-4a84-96df-be7522695d0b-workload-certs\") pod \"83287de4-d8f9-4a84-96df-be7522695d0b\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " Oct 30 13:39:07.916669 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:07.916653 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83287de4-d8f9-4a84-96df-be7522695d0b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83287de4-d8f9-4a84-96df-be7522695d0b-credential-socket\") pod \"83287de4-d8f9-4a84-96df-be7522695d0b\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " Oct 30 13:39:07.916876 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:07.916705 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83287de4-d8f9-4a84-96df-be7522695d0b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83287de4-d8f9-4a84-96df-be7522695d0b-istio-data\") pod \"83287de4-d8f9-4a84-96df-be7522695d0b\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " Oct 30 13:39:07.916876 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:07.916781 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83287de4-d8f9-4a84-96df-be7522695d0b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83287de4-d8f9-4a84-96df-be7522695d0b-istio-podinfo\") pod \"83287de4-d8f9-4a84-96df-be7522695d0b\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " Oct 30 13:39:07.916876 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:07.916852 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83287de4-d8f9-4a84-96df-be7522695d0b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83287de4-d8f9-4a84-96df-be7522695d0b-istio-envoy\") pod \"83287de4-d8f9-4a84-96df-be7522695d0b\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " Oct 30 13:39:07.916986 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:07.916893 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83287de4-d8f9-4a84-96df-be7522695d0b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83287de4-d8f9-4a84-96df-be7522695d0b-istiod-ca-cert\") pod \"83287de4-d8f9-4a84-96df-be7522695d0b\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " Oct 30 13:39:07.916986 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:07.916915 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83287de4-d8f9-4a84-96df-be7522695d0b-workload-certs" pod "83287de4-d8f9-4a84-96df-be7522695d0b" (UID: "83287de4-d8f9-4a84-96df-be7522695d0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:07.916986 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:07.916942 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83287de4-d8f9-4a84-96df-be7522695d0b-credential-socket" pod "83287de4-d8f9-4a84-96df-be7522695d0b" (UID: "83287de4-d8f9-4a84-96df-be7522695d0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:07.917116 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:07.917071 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83287de4-d8f9-4a84-96df-be7522695d0b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83287de4-d8f9-4a84-96df-be7522695d0b-workload-socket\") pod \"83287de4-d8f9-4a84-96df-be7522695d0b\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " Oct 30 13:39:07.917116 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:07.917099 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83287de4-d8f9-4a84-96df-be7522695d0b-istio-data" pod "83287de4-d8f9-4a84-96df-be7522695d0b" (UID: "83287de4-d8f9-4a84-96df-be7522695d0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:07.917242 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:07.917135 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83287de4-d8f9-4a84-96df-be7522695d0b-kube-api-access-k42fv\" (UniqueName: \"kubernetes.io/projected/83287de4-d8f9-4a84-96df-be7522695d0b-kube-api-access-k42fv\") pod \"83287de4-d8f9-4a84-96df-be7522695d0b\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " Oct 30 13:39:07.917242 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:07.917157 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83287de4-d8f9-4a84-96df-be7522695d0b-istio-token\" (UniqueName: \"kubernetes.io/projected/83287de4-d8f9-4a84-96df-be7522695d0b-istio-token\") pod \"83287de4-d8f9-4a84-96df-be7522695d0b\" (UID: \"83287de4-d8f9-4a84-96df-be7522695d0b\") " Oct 30 13:39:07.917365 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:07.917241 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83287de4-d8f9-4a84-96df-be7522695d0b-istiod-ca-cert" pod "83287de4-d8f9-4a84-96df-be7522695d0b" (UID: "83287de4-d8f9-4a84-96df-be7522695d0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:39:07.917430 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:07.917363 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83287de4-d8f9-4a84-96df-be7522695d0b-workload-socket" pod "83287de4-d8f9-4a84-96df-be7522695d0b" (UID: "83287de4-d8f9-4a84-96df-be7522695d0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:07.917792 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:07.917766 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83287de4-d8f9-4a84-96df-be7522695d0b-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:07.917941 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:07.917795 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83287de4-d8f9-4a84-96df-be7522695d0b-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:07.917941 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:07.917809 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83287de4-d8f9-4a84-96df-be7522695d0b-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:07.917941 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:07.917818 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83287de4-d8f9-4a84-96df-be7522695d0b-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:07.917941 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:07.917827 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83287de4-d8f9-4a84-96df-be7522695d0b-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:07.919108 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:07.919081 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83287de4-d8f9-4a84-96df-be7522695d0b-istio-envoy" pod "83287de4-d8f9-4a84-96df-be7522695d0b" (UID: "83287de4-d8f9-4a84-96df-be7522695d0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:07.919233 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:07.919209 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83287de4-d8f9-4a84-96df-be7522695d0b-istio-podinfo" pod "83287de4-d8f9-4a84-96df-be7522695d0b" (UID: "83287de4-d8f9-4a84-96df-be7522695d0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:39:07.919722 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:07.919700 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83287de4-d8f9-4a84-96df-be7522695d0b-istio-token" pod "83287de4-d8f9-4a84-96df-be7522695d0b" (UID: "83287de4-d8f9-4a84-96df-be7522695d0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:07.919824 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:07.919726 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83287de4-d8f9-4a84-96df-be7522695d0b-kube-api-access-k42fv" pod "83287de4-d8f9-4a84-96df-be7522695d0b" (UID: "83287de4-d8f9-4a84-96df-be7522695d0b"). InnerVolumeSpecName "kube-api-access-k42fv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:08.020643 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:08.020161 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k42fv\" (UniqueName: \"kubernetes.io/projected/83287de4-d8f9-4a84-96df-be7522695d0b-kube-api-access-k42fv\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:08.020643 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:08.020202 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83287de4-d8f9-4a84-96df-be7522695d0b-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:08.020643 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:08.020215 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83287de4-d8f9-4a84-96df-be7522695d0b-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:08.020643 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:08.020227 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83287de4-d8f9-4a84-96df-be7522695d0b-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:08.084358 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:08.080035 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn"] Oct 30 13:39:08.084358 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:08.080737 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" podUID="ec336656-bd10-4f0e-a598-5e5cd855b294" containerName="istio-proxy" containerID="cri-o://e374e69174c72d98eff478dfe35813c69cc3cd47cd59006020d4016126333ee3" gracePeriod=30 Oct 30 13:39:08.251506 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:08.251464 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft"] Oct 30 13:39:08.255121 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:08.255094 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft"] Oct 30 13:39:09.277795 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:09.277761 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-acd125-w5qhp"] Oct 30 13:39:09.291010 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:09.290977 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-acd125-w5qhp"] Oct 30 13:39:09.514458 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:09.514418 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83287de4-d8f9-4a84-96df-be7522695d0b" path="/var/lib/kubelet/pods/83287de4-d8f9-4a84-96df-be7522695d0b/volumes" Oct 30 13:39:09.514928 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:09.514914 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89b2ce06-3adf-4653-b527-2b1bf252cde6" path="/var/lib/kubelet/pods/89b2ce06-3adf-4653-b527-2b1bf252cde6/volumes" Oct 30 13:39:13.517916 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:13.517883 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" Oct 30 13:39:13.574288 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:13.570871 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec336656-bd10-4f0e-a598-5e5cd855b294-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec336656-bd10-4f0e-a598-5e5cd855b294-istio-podinfo\") pod \"ec336656-bd10-4f0e-a598-5e5cd855b294\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " Oct 30 13:39:13.574288 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:13.570949 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec336656-bd10-4f0e-a598-5e5cd855b294-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec336656-bd10-4f0e-a598-5e5cd855b294-workload-certs\") pod \"ec336656-bd10-4f0e-a598-5e5cd855b294\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " Oct 30 13:39:13.574288 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:13.570983 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec336656-bd10-4f0e-a598-5e5cd855b294-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec336656-bd10-4f0e-a598-5e5cd855b294-istiod-ca-cert\") pod \"ec336656-bd10-4f0e-a598-5e5cd855b294\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " Oct 30 13:39:13.574288 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:13.571074 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec336656-bd10-4f0e-a598-5e5cd855b294-kube-api-access-7qss9\" (UniqueName: \"kubernetes.io/projected/ec336656-bd10-4f0e-a598-5e5cd855b294-kube-api-access-7qss9\") pod \"ec336656-bd10-4f0e-a598-5e5cd855b294\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " Oct 30 13:39:13.574288 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:13.571189 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec336656-bd10-4f0e-a598-5e5cd855b294-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec336656-bd10-4f0e-a598-5e5cd855b294-istio-envoy\") pod \"ec336656-bd10-4f0e-a598-5e5cd855b294\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " Oct 30 13:39:13.574288 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:13.571226 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec336656-bd10-4f0e-a598-5e5cd855b294-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec336656-bd10-4f0e-a598-5e5cd855b294-istio-data\") pod \"ec336656-bd10-4f0e-a598-5e5cd855b294\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " Oct 30 13:39:13.574288 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:13.571435 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec336656-bd10-4f0e-a598-5e5cd855b294-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec336656-bd10-4f0e-a598-5e5cd855b294-workload-socket\") pod \"ec336656-bd10-4f0e-a598-5e5cd855b294\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " Oct 30 13:39:13.574288 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:13.571508 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec336656-bd10-4f0e-a598-5e5cd855b294-istio-token\" (UniqueName: \"kubernetes.io/projected/ec336656-bd10-4f0e-a598-5e5cd855b294-istio-token\") pod \"ec336656-bd10-4f0e-a598-5e5cd855b294\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " Oct 30 13:39:13.574288 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:13.571601 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec336656-bd10-4f0e-a598-5e5cd855b294-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec336656-bd10-4f0e-a598-5e5cd855b294-credential-socket\") pod \"ec336656-bd10-4f0e-a598-5e5cd855b294\" (UID: \"ec336656-bd10-4f0e-a598-5e5cd855b294\") " Oct 30 13:39:13.574288 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:13.572460 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec336656-bd10-4f0e-a598-5e5cd855b294-credential-socket" pod "ec336656-bd10-4f0e-a598-5e5cd855b294" (UID: "ec336656-bd10-4f0e-a598-5e5cd855b294"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:13.574288 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:13.573461 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec336656-bd10-4f0e-a598-5e5cd855b294-workload-certs" pod "ec336656-bd10-4f0e-a598-5e5cd855b294" (UID: "ec336656-bd10-4f0e-a598-5e5cd855b294"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:13.574288 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:13.573634 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec336656-bd10-4f0e-a598-5e5cd855b294-workload-socket" pod "ec336656-bd10-4f0e-a598-5e5cd855b294" (UID: "ec336656-bd10-4f0e-a598-5e5cd855b294"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:13.574288 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:13.573804 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec336656-bd10-4f0e-a598-5e5cd855b294-istio-data" pod "ec336656-bd10-4f0e-a598-5e5cd855b294" (UID: "ec336656-bd10-4f0e-a598-5e5cd855b294"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:13.574288 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:13.574199 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec336656-bd10-4f0e-a598-5e5cd855b294-istiod-ca-cert" pod "ec336656-bd10-4f0e-a598-5e5cd855b294" (UID: "ec336656-bd10-4f0e-a598-5e5cd855b294"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:39:13.599436 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:13.599407 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec336656-bd10-4f0e-a598-5e5cd855b294-istio-podinfo" pod "ec336656-bd10-4f0e-a598-5e5cd855b294" (UID: "ec336656-bd10-4f0e-a598-5e5cd855b294"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:39:13.610097 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:13.607473 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec336656-bd10-4f0e-a598-5e5cd855b294-istio-envoy" pod "ec336656-bd10-4f0e-a598-5e5cd855b294" (UID: "ec336656-bd10-4f0e-a598-5e5cd855b294"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:13.610097 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:13.607695 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec336656-bd10-4f0e-a598-5e5cd855b294-kube-api-access-7qss9" pod "ec336656-bd10-4f0e-a598-5e5cd855b294" (UID: "ec336656-bd10-4f0e-a598-5e5cd855b294"). InnerVolumeSpecName "kube-api-access-7qss9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:13.610097 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:13.608693 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec336656-bd10-4f0e-a598-5e5cd855b294-istio-token" pod "ec336656-bd10-4f0e-a598-5e5cd855b294" (UID: "ec336656-bd10-4f0e-a598-5e5cd855b294"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:13.673685 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:13.673648 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec336656-bd10-4f0e-a598-5e5cd855b294-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:13.673685 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:13.673682 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec336656-bd10-4f0e-a598-5e5cd855b294-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:13.673921 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:13.673695 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qss9\" (UniqueName: \"kubernetes.io/projected/ec336656-bd10-4f0e-a598-5e5cd855b294-kube-api-access-7qss9\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:13.673921 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:13.673708 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec336656-bd10-4f0e-a598-5e5cd855b294-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:13.673921 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:13.673721 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec336656-bd10-4f0e-a598-5e5cd855b294-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:13.673921 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:13.673732 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec336656-bd10-4f0e-a598-5e5cd855b294-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:13.673921 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:13.673742 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec336656-bd10-4f0e-a598-5e5cd855b294-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:13.673921 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:13.673754 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec336656-bd10-4f0e-a598-5e5cd855b294-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:13.673921 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:13.673766 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec336656-bd10-4f0e-a598-5e5cd855b294-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:14.000379 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:13.895056 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n"] Oct 30 13:39:14.001753 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:14.001710 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n" podUID="f1342efa-24d9-414d-ac5b-c4efc78ce33c" containerName="maas-api" containerID="cri-o://fac285f6638d54dc8f1c242f97d82d6d977fee4a8fb551aa0619b245ee3291c6" gracePeriod=30 Oct 30 13:39:14.085064 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:14.085020 4931 generic.go:356] "Generic (PLEG): container finished" podID="ec336656-bd10-4f0e-a598-5e5cd855b294" containerID="e374e69174c72d98eff478dfe35813c69cc3cd47cd59006020d4016126333ee3" exitCode=0 Oct 30 13:39:14.085292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:14.085254 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" event={"ID":"ec336656-bd10-4f0e-a598-5e5cd855b294","Type":"ContainerDied","Data":"e374e69174c72d98eff478dfe35813c69cc3cd47cd59006020d4016126333ee3"} Oct 30 13:39:14.085391 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:14.085333 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" event={"ID":"ec336656-bd10-4f0e-a598-5e5cd855b294","Type":"ContainerDied","Data":"11735ca017a4667443b33217353abaf53db26c95bf8d8b941f3e6b00b767b892"} Oct 30 13:39:14.085391 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:14.085365 4931 scope.go:122] "RemoveContainer" containerID="e374e69174c72d98eff478dfe35813c69cc3cd47cd59006020d4016126333ee3" Oct 30 13:39:14.085585 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:14.085568 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn" Oct 30 13:39:14.335508 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:14.335436 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn"] Oct 30 13:39:14.336707 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:14.335948 4931 scope.go:122] "RemoveContainer" containerID="e374e69174c72d98eff478dfe35813c69cc3cd47cd59006020d4016126333ee3" Oct 30 13:39:14.355030 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:14.341355 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn"] Oct 30 13:39:14.355030 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:39:14.346659 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e374e69174c72d98eff478dfe35813c69cc3cd47cd59006020d4016126333ee3\": container with ID starting with e374e69174c72d98eff478dfe35813c69cc3cd47cd59006020d4016126333ee3 not found: ID does not exist" containerID="e374e69174c72d98eff478dfe35813c69cc3cd47cd59006020d4016126333ee3" Oct 30 13:39:14.355030 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:14.346700 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e374e69174c72d98eff478dfe35813c69cc3cd47cd59006020d4016126333ee3"} err="failed to get container status \"e374e69174c72d98eff478dfe35813c69cc3cd47cd59006020d4016126333ee3\": rpc error: code = NotFound desc = could not find container \"e374e69174c72d98eff478dfe35813c69cc3cd47cd59006020d4016126333ee3\": container with ID starting with e374e69174c72d98eff478dfe35813c69cc3cd47cd59006020d4016126333ee3 not found: ID does not exist" Oct 30 13:39:14.539362 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:14.528319 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c91909-4rb57"] Oct 30 13:39:14.613986 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:14.612732 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c91909-4rb57"] Oct 30 13:39:14.769746 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:14.769719 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n" Oct 30 13:39:14.855216 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:14.855128 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1342efa-24d9-414d-ac5b-c4efc78ce33c-kube-api-access-rn8m7\" (UniqueName: \"kubernetes.io/projected/f1342efa-24d9-414d-ac5b-c4efc78ce33c-kube-api-access-rn8m7\") pod \"f1342efa-24d9-414d-ac5b-c4efc78ce33c\" (UID: \"f1342efa-24d9-414d-ac5b-c4efc78ce33c\") " Oct 30 13:39:14.855216 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:14.855179 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1342efa-24d9-414d-ac5b-c4efc78ce33c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1342efa-24d9-414d-ac5b-c4efc78ce33c-openshift-service-ca\") pod \"f1342efa-24d9-414d-ac5b-c4efc78ce33c\" (UID: \"f1342efa-24d9-414d-ac5b-c4efc78ce33c\") " Oct 30 13:39:14.855512 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:14.855424 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1342efa-24d9-414d-ac5b-c4efc78ce33c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1342efa-24d9-414d-ac5b-c4efc78ce33c-metrics-certs\") pod \"f1342efa-24d9-414d-ac5b-c4efc78ce33c\" (UID: \"f1342efa-24d9-414d-ac5b-c4efc78ce33c\") " Oct 30 13:39:14.855512 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:14.855454 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1342efa-24d9-414d-ac5b-c4efc78ce33c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1342efa-24d9-414d-ac5b-c4efc78ce33c-maas-api-tls\") pod \"f1342efa-24d9-414d-ac5b-c4efc78ce33c\" (UID: \"f1342efa-24d9-414d-ac5b-c4efc78ce33c\") " Oct 30 13:39:14.855660 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:14.855624 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1342efa-24d9-414d-ac5b-c4efc78ce33c-openshift-service-ca" pod "f1342efa-24d9-414d-ac5b-c4efc78ce33c" (UID: "f1342efa-24d9-414d-ac5b-c4efc78ce33c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:39:14.856129 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:14.856102 4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1342efa-24d9-414d-ac5b-c4efc78ce33c-openshift-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:14.858070 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:14.858020 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1342efa-24d9-414d-ac5b-c4efc78ce33c-metrics-certs" pod "f1342efa-24d9-414d-ac5b-c4efc78ce33c" (UID: "f1342efa-24d9-414d-ac5b-c4efc78ce33c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:39:14.858203 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:14.858126 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1342efa-24d9-414d-ac5b-c4efc78ce33c-maas-api-tls" pod "f1342efa-24d9-414d-ac5b-c4efc78ce33c" (UID: "f1342efa-24d9-414d-ac5b-c4efc78ce33c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:39:14.858203 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:14.858130 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1342efa-24d9-414d-ac5b-c4efc78ce33c-kube-api-access-rn8m7" pod "f1342efa-24d9-414d-ac5b-c4efc78ce33c" (UID: "f1342efa-24d9-414d-ac5b-c4efc78ce33c"). InnerVolumeSpecName "kube-api-access-rn8m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:14.957890 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:14.957858 4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1342efa-24d9-414d-ac5b-c4efc78ce33c-metrics-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:14.957890 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:14.957885 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1342efa-24d9-414d-ac5b-c4efc78ce33c-maas-api-tls\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:14.957890 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:14.957896 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn8m7\" (UniqueName: \"kubernetes.io/projected/f1342efa-24d9-414d-ac5b-c4efc78ce33c-kube-api-access-rn8m7\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:15.069519 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:15.067164 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2afe58-g92hc"] Oct 30 13:39:15.070918 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:15.070874 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2afe58-g92hc"] Oct 30 13:39:15.094329 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:15.094295 4931 generic.go:356] "Generic (PLEG): container finished" podID="f1342efa-24d9-414d-ac5b-c4efc78ce33c" containerID="fac285f6638d54dc8f1c242f97d82d6d977fee4a8fb551aa0619b245ee3291c6" exitCode=0 Oct 30 13:39:15.094329 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:15.094324 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n" Oct 30 13:39:15.094526 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:15.094384 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n" event={"ID":"f1342efa-24d9-414d-ac5b-c4efc78ce33c","Type":"ContainerDied","Data":"fac285f6638d54dc8f1c242f97d82d6d977fee4a8fb551aa0619b245ee3291c6"} Oct 30 13:39:15.094526 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:15.094437 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n" event={"ID":"f1342efa-24d9-414d-ac5b-c4efc78ce33c","Type":"ContainerDied","Data":"6a2e17819d52a3afc74a5fd341010e123bbc2412debd481fc43e17c494960bcb"} Oct 30 13:39:15.094526 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:15.094463 4931 scope.go:122] "RemoveContainer" containerID="fac285f6638d54dc8f1c242f97d82d6d977fee4a8fb551aa0619b245ee3291c6" Oct 30 13:39:15.107246 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:15.107225 4931 scope.go:122] "RemoveContainer" containerID="fac285f6638d54dc8f1c242f97d82d6d977fee4a8fb551aa0619b245ee3291c6" Oct 30 13:39:15.107601 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:39:15.107575 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fac285f6638d54dc8f1c242f97d82d6d977fee4a8fb551aa0619b245ee3291c6\": container with ID starting with fac285f6638d54dc8f1c242f97d82d6d977fee4a8fb551aa0619b245ee3291c6 not found: ID does not exist" containerID="fac285f6638d54dc8f1c242f97d82d6d977fee4a8fb551aa0619b245ee3291c6" Oct 30 13:39:15.107690 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:15.107613 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fac285f6638d54dc8f1c242f97d82d6d977fee4a8fb551aa0619b245ee3291c6"} err="failed to get container status \"fac285f6638d54dc8f1c242f97d82d6d977fee4a8fb551aa0619b245ee3291c6\": rpc error: code = NotFound desc = could not find container \"fac285f6638d54dc8f1c242f97d82d6d977fee4a8fb551aa0619b245ee3291c6\": container with ID starting with fac285f6638d54dc8f1c242f97d82d6d977fee4a8fb551aa0619b245ee3291c6 not found: ID does not exist" Oct 30 13:39:15.154737 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:15.154709 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n"] Oct 30 13:39:15.159442 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:15.159419 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2afe58-5cdf6c6f7-4gf6n"] Oct 30 13:39:15.513350 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:15.513313 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ba37aa6-09b2-4e31-a7f4-86db3413c202" path="/var/lib/kubelet/pods/6ba37aa6-09b2-4e31-a7f4-86db3413c202/volumes" Oct 30 13:39:15.513976 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:15.513951 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec336656-bd10-4f0e-a598-5e5cd855b294" path="/var/lib/kubelet/pods/ec336656-bd10-4f0e-a598-5e5cd855b294/volumes" Oct 30 13:39:15.514479 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:15.514458 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1342efa-24d9-414d-ac5b-c4efc78ce33c" path="/var/lib/kubelet/pods/f1342efa-24d9-414d-ac5b-c4efc78ce33c/volumes" Oct 30 13:39:15.515236 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:15.515215 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd73f227-55d8-4861-9845-1586fd021731" path="/var/lib/kubelet/pods/fd73f227-55d8-4861-9845-1586fd021731/volumes" Oct 30 13:39:16.651245 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:16.651210 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs"] Oct 30 13:39:16.651698 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:16.651428 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs" podUID="7f409e7f-6b24-410e-896c-a8a557298edf" containerName="maas-api" containerID="cri-o://a689cf489f9c72ecb3333b4ac24a4eab30a69cfa7f3a8d80c9efd2d02f690988" gracePeriod=30 Oct 30 13:39:16.995572 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:16.995525 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs" Oct 30 13:39:17.109846 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:17.095793 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f409e7f-6b24-410e-896c-a8a557298edf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f409e7f-6b24-410e-896c-a8a557298edf-metrics-certs\") pod \"7f409e7f-6b24-410e-896c-a8a557298edf\" (UID: \"7f409e7f-6b24-410e-896c-a8a557298edf\") " Oct 30 13:39:17.109846 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:17.095885 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f409e7f-6b24-410e-896c-a8a557298edf-kube-api-access-d6x6s\" (UniqueName: \"kubernetes.io/projected/7f409e7f-6b24-410e-896c-a8a557298edf-kube-api-access-d6x6s\") pod \"7f409e7f-6b24-410e-896c-a8a557298edf\" (UID: \"7f409e7f-6b24-410e-896c-a8a557298edf\") " Oct 30 13:39:17.109846 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:17.095928 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f409e7f-6b24-410e-896c-a8a557298edf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f409e7f-6b24-410e-896c-a8a557298edf-openshift-service-ca\") pod \"7f409e7f-6b24-410e-896c-a8a557298edf\" (UID: \"7f409e7f-6b24-410e-896c-a8a557298edf\") " Oct 30 13:39:17.109846 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:17.096038 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f409e7f-6b24-410e-896c-a8a557298edf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f409e7f-6b24-410e-896c-a8a557298edf-maas-api-tls\") pod \"7f409e7f-6b24-410e-896c-a8a557298edf\" (UID: \"7f409e7f-6b24-410e-896c-a8a557298edf\") " Oct 30 13:39:17.111562 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:17.111505 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f409e7f-6b24-410e-896c-a8a557298edf-openshift-service-ca" pod "7f409e7f-6b24-410e-896c-a8a557298edf" (UID: "7f409e7f-6b24-410e-896c-a8a557298edf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:39:17.112993 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:17.112925 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f409e7f-6b24-410e-896c-a8a557298edf-maas-api-tls" pod "7f409e7f-6b24-410e-896c-a8a557298edf" (UID: "7f409e7f-6b24-410e-896c-a8a557298edf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:39:17.116653 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:17.116140 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f409e7f-6b24-410e-896c-a8a557298edf-kube-api-access-d6x6s" pod "7f409e7f-6b24-410e-896c-a8a557298edf" (UID: "7f409e7f-6b24-410e-896c-a8a557298edf"). InnerVolumeSpecName "kube-api-access-d6x6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:17.116911 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:17.116881 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f409e7f-6b24-410e-896c-a8a557298edf-metrics-certs" pod "7f409e7f-6b24-410e-896c-a8a557298edf" (UID: "7f409e7f-6b24-410e-896c-a8a557298edf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:39:17.126903 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:17.126501 4931 generic.go:356] "Generic (PLEG): container finished" podID="7f409e7f-6b24-410e-896c-a8a557298edf" containerID="a689cf489f9c72ecb3333b4ac24a4eab30a69cfa7f3a8d80c9efd2d02f690988" exitCode=0 Oct 30 13:39:17.126903 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:17.126702 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs" event={"ID":"7f409e7f-6b24-410e-896c-a8a557298edf","Type":"ContainerDied","Data":"a689cf489f9c72ecb3333b4ac24a4eab30a69cfa7f3a8d80c9efd2d02f690988"} Oct 30 13:39:17.126903 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:17.126756 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs" event={"ID":"7f409e7f-6b24-410e-896c-a8a557298edf","Type":"ContainerDied","Data":"91f42af699552dd2c69ade4dc10b9dc5aedfad8a6311dbe843c7774c6e2f92dd"} Oct 30 13:39:17.126903 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:17.126787 4931 scope.go:122] "RemoveContainer" containerID="a689cf489f9c72ecb3333b4ac24a4eab30a69cfa7f3a8d80c9efd2d02f690988" Oct 30 13:39:17.127192 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:17.126957 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs" Oct 30 13:39:17.146762 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:17.146742 4931 scope.go:122] "RemoveContainer" containerID="a689cf489f9c72ecb3333b4ac24a4eab30a69cfa7f3a8d80c9efd2d02f690988" Oct 30 13:39:17.147235 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:39:17.147129 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a689cf489f9c72ecb3333b4ac24a4eab30a69cfa7f3a8d80c9efd2d02f690988\": container with ID starting with a689cf489f9c72ecb3333b4ac24a4eab30a69cfa7f3a8d80c9efd2d02f690988 not found: ID does not exist" containerID="a689cf489f9c72ecb3333b4ac24a4eab30a69cfa7f3a8d80c9efd2d02f690988" Oct 30 13:39:17.147235 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:17.147169 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a689cf489f9c72ecb3333b4ac24a4eab30a69cfa7f3a8d80c9efd2d02f690988"} err="failed to get container status \"a689cf489f9c72ecb3333b4ac24a4eab30a69cfa7f3a8d80c9efd2d02f690988\": rpc error: code = NotFound desc = could not find container \"a689cf489f9c72ecb3333b4ac24a4eab30a69cfa7f3a8d80c9efd2d02f690988\": container with ID starting with a689cf489f9c72ecb3333b4ac24a4eab30a69cfa7f3a8d80c9efd2d02f690988 not found: ID does not exist" Oct 30 13:39:17.196550 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:17.196520 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs"] Oct 30 13:39:17.198421 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:17.198347 4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f409e7f-6b24-410e-896c-a8a557298edf-metrics-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:17.198421 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:17.198372 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6x6s\" (UniqueName: \"kubernetes.io/projected/7f409e7f-6b24-410e-896c-a8a557298edf-kube-api-access-d6x6s\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:17.198421 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:17.198384 4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f409e7f-6b24-410e-896c-a8a557298edf-openshift-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:17.198421 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:17.198393 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f409e7f-6b24-410e-896c-a8a557298edf-maas-api-tls\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:17.219020 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:17.218985 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-72208f-574ffcb867-5wzgs"] Oct 30 13:39:17.513999 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:17.513903 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f409e7f-6b24-410e-896c-a8a557298edf" path="/var/lib/kubelet/pods/7f409e7f-6b24-410e-896c-a8a557298edf/volumes" Oct 30 13:39:17.536707 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:17.536674 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-mq6wx"] Oct 30 13:39:17.541592 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:17.541555 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-mq6wx"] Oct 30 13:39:17.827830 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:17.827730 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx"] Oct 30 13:39:17.829017 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:17.828973 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" podUID="0f2721af-325b-407d-8e1e-c1ed692291e8" containerName="istio-proxy" containerID="cri-o://21e09ea257786f910869b4a2138a4b0d8977f6fc51a882f77c64709d4224f27b" gracePeriod=30 Oct 30 13:39:18.757651 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.757614 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-96ab73-qxttd"] Oct 30 13:39:18.769876 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.769841 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-96ab73-qxttd"] Oct 30 13:39:18.835669 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.835631 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e9d641-rhwl7"] Oct 30 13:39:18.837584 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.837550 4931 cpu_manager.go:414] "containerMap: removing container" podUID="89b2ce06-3adf-4653-b527-2b1bf252cde6" containerName="revoke-keys" Oct 30 13:39:18.837584 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.837582 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89b2ce06-3adf-4653-b527-2b1bf252cde6" containerName="revoke-keys" Oct 30 13:39:18.837835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.837612 4931 cpu_manager.go:414] "containerMap: removing container" podUID="5c6dbc7e-8785-4971-88c4-e3341faaadff" containerName="istio-proxy" Oct 30 13:39:18.837835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.837620 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c6dbc7e-8785-4971-88c4-e3341faaadff" containerName="istio-proxy" Oct 30 13:39:18.837835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.837631 4931 cpu_manager.go:414] "containerMap: removing container" podUID="6ba37aa6-09b2-4e31-a7f4-86db3413c202" containerName="revoke-keys" Oct 30 13:39:18.837835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.837637 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ba37aa6-09b2-4e31-a7f4-86db3413c202" containerName="revoke-keys" Oct 30 13:39:18.837835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.837655 4931 cpu_manager.go:414] "containerMap: removing container" podUID="ec336656-bd10-4f0e-a598-5e5cd855b294" containerName="istio-proxy" Oct 30 13:39:18.837835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.837663 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec336656-bd10-4f0e-a598-5e5cd855b294" containerName="istio-proxy" Oct 30 13:39:18.837835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.837687 4931 cpu_manager.go:414] "containerMap: removing container" podUID="813ffc2b-3c94-4847-aa9b-2d7971da6e4d" containerName="istio-proxy" Oct 30 13:39:18.837835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.837695 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="813ffc2b-3c94-4847-aa9b-2d7971da6e4d" containerName="istio-proxy" Oct 30 13:39:18.837835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.837711 4931 cpu_manager.go:414] "containerMap: removing container" podUID="c666cfb4-8b66-49b3-b248-2f2771088444" containerName="istio-proxy" Oct 30 13:39:18.837835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.837720 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c666cfb4-8b66-49b3-b248-2f2771088444" containerName="istio-proxy" Oct 30 13:39:18.837835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.837737 4931 cpu_manager.go:414] "containerMap: removing container" podUID="9916ba3d-3bef-4338-aa2e-f5c27782188f" containerName="istio-proxy" Oct 30 13:39:18.837835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.837743 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9916ba3d-3bef-4338-aa2e-f5c27782188f" containerName="istio-proxy" Oct 30 13:39:18.837835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.837762 4931 cpu_manager.go:414] "containerMap: removing container" podUID="83287de4-d8f9-4a84-96df-be7522695d0b" containerName="istio-proxy" Oct 30 13:39:18.837835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.837769 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83287de4-d8f9-4a84-96df-be7522695d0b" containerName="istio-proxy" Oct 30 13:39:18.837835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.837787 4931 cpu_manager.go:414] "containerMap: removing container" podUID="f1342efa-24d9-414d-ac5b-c4efc78ce33c" containerName="maas-api" Oct 30 13:39:18.837835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.837795 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1342efa-24d9-414d-ac5b-c4efc78ce33c" containerName="maas-api" Oct 30 13:39:18.837835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.837805 4931 cpu_manager.go:414] "containerMap: removing container" podUID="d6f71a7e-fbeb-4e16-8863-e17fc9695553" containerName="istio-proxy" Oct 30 13:39:18.837835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.837813 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6f71a7e-fbeb-4e16-8863-e17fc9695553" containerName="istio-proxy" Oct 30 13:39:18.837835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.837845 4931 cpu_manager.go:414] "containerMap: removing container" podUID="6a38c538-ca9f-4135-96f8-32dff99e07d7" containerName="istio-proxy" Oct 30 13:39:18.838842 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.837854 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a38c538-ca9f-4135-96f8-32dff99e07d7" containerName="istio-proxy" Oct 30 13:39:18.838842 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.837871 4931 cpu_manager.go:414] "containerMap: removing container" podUID="7f409e7f-6b24-410e-896c-a8a557298edf" containerName="maas-api" Oct 30 13:39:18.838842 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.837879 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f409e7f-6b24-410e-896c-a8a557298edf" containerName="maas-api" Oct 30 13:39:18.838842 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.838068 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f409e7f-6b24-410e-896c-a8a557298edf" containerName="maas-api" Oct 30 13:39:18.838842 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.838086 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="89b2ce06-3adf-4653-b527-2b1bf252cde6" containerName="revoke-keys" Oct 30 13:39:18.838842 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.838097 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c6dbc7e-8785-4971-88c4-e3341faaadff" containerName="istio-proxy" Oct 30 13:39:18.838842 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.838107 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ba37aa6-09b2-4e31-a7f4-86db3413c202" containerName="revoke-keys" Oct 30 13:39:18.838842 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.838117 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec336656-bd10-4f0e-a598-5e5cd855b294" containerName="istio-proxy" Oct 30 13:39:18.838842 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.838127 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="813ffc2b-3c94-4847-aa9b-2d7971da6e4d" containerName="istio-proxy" Oct 30 13:39:18.838842 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.838135 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="c666cfb4-8b66-49b3-b248-2f2771088444" containerName="istio-proxy" Oct 30 13:39:18.838842 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.838144 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="9916ba3d-3bef-4338-aa2e-f5c27782188f" containerName="istio-proxy" Oct 30 13:39:18.838842 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.838153 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="83287de4-d8f9-4a84-96df-be7522695d0b" containerName="istio-proxy" Oct 30 13:39:18.838842 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.838163 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1342efa-24d9-414d-ac5b-c4efc78ce33c" containerName="maas-api" Oct 30 13:39:18.838842 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.838172 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6f71a7e-fbeb-4e16-8863-e17fc9695553" containerName="istio-proxy" Oct 30 13:39:18.838842 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.838183 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a38c538-ca9f-4135-96f8-32dff99e07d7" containerName="istio-proxy" Oct 30 13:39:18.843444 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.843418 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e9d641-rhwl7" Oct 30 13:39:18.863318 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.863289 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e9d641-rhwl7"] Oct 30 13:39:18.881509 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.881473 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:39:18.881700 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.881511 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:39:18.938634 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.938599 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vkzw\" (UniqueName: \"kubernetes.io/projected/67e16bff-a9c9-42d1-b787-dd6f0f98658d-kube-api-access-5vkzw\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e9d641-rhwl7\" (UID: \"67e16bff-a9c9-42d1-b787-dd6f0f98658d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e9d641-rhwl7" Oct 30 13:39:18.938844 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.938680 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67e16bff-a9c9-42d1-b787-dd6f0f98658d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e9d641-rhwl7\" (UID: \"67e16bff-a9c9-42d1-b787-dd6f0f98658d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e9d641-rhwl7" Oct 30 13:39:18.938844 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:18.938779 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67e16bff-a9c9-42d1-b787-dd6f0f98658d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e9d641-rhwl7\" (UID: \"67e16bff-a9c9-42d1-b787-dd6f0f98658d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e9d641-rhwl7" Oct 30 13:39:19.040576 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:19.040485 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67e16bff-a9c9-42d1-b787-dd6f0f98658d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e9d641-rhwl7\" (UID: \"67e16bff-a9c9-42d1-b787-dd6f0f98658d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e9d641-rhwl7" Oct 30 13:39:19.040576 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:19.040539 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67e16bff-a9c9-42d1-b787-dd6f0f98658d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e9d641-rhwl7\" (UID: \"67e16bff-a9c9-42d1-b787-dd6f0f98658d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e9d641-rhwl7" Oct 30 13:39:19.040828 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:19.040753 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vkzw\" (UniqueName: \"kubernetes.io/projected/67e16bff-a9c9-42d1-b787-dd6f0f98658d-kube-api-access-5vkzw\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e9d641-rhwl7\" (UID: \"67e16bff-a9c9-42d1-b787-dd6f0f98658d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e9d641-rhwl7" Oct 30 13:39:19.041355 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:19.041329 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67e16bff-a9c9-42d1-b787-dd6f0f98658d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e9d641-rhwl7\" (UID: \"67e16bff-a9c9-42d1-b787-dd6f0f98658d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e9d641-rhwl7" Oct 30 13:39:19.042945 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:19.042921 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67e16bff-a9c9-42d1-b787-dd6f0f98658d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e9d641-rhwl7\" (UID: \"67e16bff-a9c9-42d1-b787-dd6f0f98658d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e9d641-rhwl7" Oct 30 13:39:19.065645 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:19.065622 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vkzw\" (UniqueName: \"kubernetes.io/projected/67e16bff-a9c9-42d1-b787-dd6f0f98658d-kube-api-access-5vkzw\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e9d641-rhwl7\" (UID: \"67e16bff-a9c9-42d1-b787-dd6f0f98658d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e9d641-rhwl7" Oct 30 13:39:19.159489 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:19.159456 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e9d641-rhwl7" Oct 30 13:39:19.466621 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:19.466589 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e9d641-rhwl7"] Oct 30 13:39:19.467580 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:39:19.467547 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67e16bff_a9c9_42d1_b787_dd6f0f98658d.slice/crio-909351ff0c64b9100ea915c427c85182ff2445c404efbf2c392c31e74c418097 WatchSource:0}: Error finding container 909351ff0c64b9100ea915c427c85182ff2445c404efbf2c392c31e74c418097: Status 404 returned error can't find the container with id 909351ff0c64b9100ea915c427c85182ff2445c404efbf2c392c31e74c418097 Oct 30 13:39:19.515349 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:19.515308 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66b6003e-28c2-4275-9b9c-6865fd5f7b60" path="/var/lib/kubelet/pods/66b6003e-28c2-4275-9b9c-6865fd5f7b60/volumes" Oct 30 13:39:19.515907 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:19.515884 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deea9d6c-9b3a-40d5-9b90-74818449aed0" path="/var/lib/kubelet/pods/deea9d6c-9b3a-40d5-9b90-74818449aed0/volumes" Oct 30 13:39:19.980204 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:19.980168 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw"] Oct 30 13:39:19.980644 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:19.980413 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" podUID="db26d540-4441-49aa-be9f-51edab84b99c" containerName="istio-proxy" containerID="cri-o://5818472411fbdb6fe85ea3df7c0044ccb9f4f25d61bf7fe97baec779d9a1e93a" gracePeriod=30 Oct 30 13:39:20.153329 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:20.153197 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e9d641-rhwl7" event={"ID":"67e16bff-a9c9-42d1-b787-dd6f0f98658d","Type":"ContainerStarted","Data":"92a2ae4dd38ea66b970b8365af6b30d2f2834bf3e25a4d8c57b2e82b44ccd3ce"} Oct 30 13:39:20.153329 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:20.153316 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e9d641-rhwl7" event={"ID":"67e16bff-a9c9-42d1-b787-dd6f0f98658d","Type":"ContainerStarted","Data":"909351ff0c64b9100ea915c427c85182ff2445c404efbf2c392c31e74c418097"} Oct 30 13:39:20.205124 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:20.205072 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e9d641-rhwl7" podStartSLOduration=2.205058691 podStartE2EDuration="2.205058691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:39:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:39:20.203298271 +0000 UTC m=+3258.806796718" watchObservedRunningTime="2026-10-30 13:39:20.205058691 +0000 UTC m=+3258.808557137" Oct 30 13:39:21.163240 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:21.163202 4931 generic.go:356] "Generic (PLEG): container finished" podID="67e16bff-a9c9-42d1-b787-dd6f0f98658d" containerID="92a2ae4dd38ea66b970b8365af6b30d2f2834bf3e25a4d8c57b2e82b44ccd3ce" exitCode=0 Oct 30 13:39:21.163726 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:21.163317 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e9d641-rhwl7" event={"ID":"67e16bff-a9c9-42d1-b787-dd6f0f98658d","Type":"ContainerDied","Data":"92a2ae4dd38ea66b970b8365af6b30d2f2834bf3e25a4d8c57b2e82b44ccd3ce"} Oct 30 13:39:22.328134 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:22.328103 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e9d641-rhwl7" Oct 30 13:39:22.386089 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:22.386056 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67e16bff-a9c9-42d1-b787-dd6f0f98658d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67e16bff-a9c9-42d1-b787-dd6f0f98658d-maas-api-xks-ca\") pod \"67e16bff-a9c9-42d1-b787-dd6f0f98658d\" (UID: \"67e16bff-a9c9-42d1-b787-dd6f0f98658d\") " Oct 30 13:39:22.386248 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:22.386162 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67e16bff-a9c9-42d1-b787-dd6f0f98658d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67e16bff-a9c9-42d1-b787-dd6f0f98658d-maas-api-openshift-ca\") pod \"67e16bff-a9c9-42d1-b787-dd6f0f98658d\" (UID: \"67e16bff-a9c9-42d1-b787-dd6f0f98658d\") " Oct 30 13:39:22.386248 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:22.386217 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67e16bff-a9c9-42d1-b787-dd6f0f98658d-kube-api-access-5vkzw\" (UniqueName: \"kubernetes.io/projected/67e16bff-a9c9-42d1-b787-dd6f0f98658d-kube-api-access-5vkzw\") pod \"67e16bff-a9c9-42d1-b787-dd6f0f98658d\" (UID: \"67e16bff-a9c9-42d1-b787-dd6f0f98658d\") " Oct 30 13:39:22.386538 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:22.386512 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67e16bff-a9c9-42d1-b787-dd6f0f98658d-maas-api-openshift-ca" pod "67e16bff-a9c9-42d1-b787-dd6f0f98658d" (UID: "67e16bff-a9c9-42d1-b787-dd6f0f98658d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:39:22.386831 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:22.386804 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67e16bff-a9c9-42d1-b787-dd6f0f98658d-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:22.388576 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:22.388551 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e16bff-a9c9-42d1-b787-dd6f0f98658d-kube-api-access-5vkzw" pod "67e16bff-a9c9-42d1-b787-dd6f0f98658d" (UID: "67e16bff-a9c9-42d1-b787-dd6f0f98658d"). InnerVolumeSpecName "kube-api-access-5vkzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:22.388700 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:22.388618 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67e16bff-a9c9-42d1-b787-dd6f0f98658d-maas-api-xks-ca" pod "67e16bff-a9c9-42d1-b787-dd6f0f98658d" (UID: "67e16bff-a9c9-42d1-b787-dd6f0f98658d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:39:22.488053 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:22.488023 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67e16bff-a9c9-42d1-b787-dd6f0f98658d-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:22.488053 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:22.488052 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vkzw\" (UniqueName: \"kubernetes.io/projected/67e16bff-a9c9-42d1-b787-dd6f0f98658d-kube-api-access-5vkzw\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:23.232629 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.232591 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e9d641-rhwl7" event={"ID":"67e16bff-a9c9-42d1-b787-dd6f0f98658d","Type":"ContainerDied","Data":"909351ff0c64b9100ea915c427c85182ff2445c404efbf2c392c31e74c418097"} Oct 30 13:39:23.232867 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.232852 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="909351ff0c64b9100ea915c427c85182ff2445c404efbf2c392c31e74c418097" Oct 30 13:39:23.233050 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.233039 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e9d641-rhwl7" Oct 30 13:39:23.238554 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.238237 4931 generic.go:356] "Generic (PLEG): container finished" podID="0f2721af-325b-407d-8e1e-c1ed692291e8" containerID="21e09ea257786f910869b4a2138a4b0d8977f6fc51a882f77c64709d4224f27b" exitCode=0 Oct 30 13:39:23.238554 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.238364 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" event={"ID":"0f2721af-325b-407d-8e1e-c1ed692291e8","Type":"ContainerDied","Data":"21e09ea257786f910869b4a2138a4b0d8977f6fc51a882f77c64709d4224f27b"} Oct 30 13:39:23.238554 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.238403 4931 scope.go:122] "RemoveContainer" containerID="ccf15ff02cc32911aaa6ab110724aee34fd6f8c45497292b7c656b93f277e9be" Oct 30 13:39:23.446220 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.446191 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:39:23.534863 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.534782 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f2721af-325b-407d-8e1e-c1ed692291e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2721af-325b-407d-8e1e-c1ed692291e8-credential-socket\") pod \"0f2721af-325b-407d-8e1e-c1ed692291e8\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " Oct 30 13:39:23.534863 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.534854 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f2721af-325b-407d-8e1e-c1ed692291e8-kube-api-access-xrl4f\" (UniqueName: \"kubernetes.io/projected/0f2721af-325b-407d-8e1e-c1ed692291e8-kube-api-access-xrl4f\") pod \"0f2721af-325b-407d-8e1e-c1ed692291e8\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " Oct 30 13:39:23.535090 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.534924 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f2721af-325b-407d-8e1e-c1ed692291e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f2721af-325b-407d-8e1e-c1ed692291e8-workload-certs\") pod \"0f2721af-325b-407d-8e1e-c1ed692291e8\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " Oct 30 13:39:23.535090 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.535005 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f2721af-325b-407d-8e1e-c1ed692291e8-istio-token\" (UniqueName: \"kubernetes.io/projected/0f2721af-325b-407d-8e1e-c1ed692291e8-istio-token\") pod \"0f2721af-325b-407d-8e1e-c1ed692291e8\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " Oct 30 13:39:23.535090 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.535048 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f2721af-325b-407d-8e1e-c1ed692291e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f2721af-325b-407d-8e1e-c1ed692291e8-istio-podinfo\") pod \"0f2721af-325b-407d-8e1e-c1ed692291e8\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " Oct 30 13:39:23.535090 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.535085 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f2721af-325b-407d-8e1e-c1ed692291e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f2721af-325b-407d-8e1e-c1ed692291e8-istiod-ca-cert\") pod \"0f2721af-325b-407d-8e1e-c1ed692291e8\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " Oct 30 13:39:23.535331 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.535144 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f2721af-325b-407d-8e1e-c1ed692291e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f2721af-325b-407d-8e1e-c1ed692291e8-istio-data\") pod \"0f2721af-325b-407d-8e1e-c1ed692291e8\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " Oct 30 13:39:23.535331 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.535219 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f2721af-325b-407d-8e1e-c1ed692291e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2721af-325b-407d-8e1e-c1ed692291e8-workload-socket\") pod \"0f2721af-325b-407d-8e1e-c1ed692291e8\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " Oct 30 13:39:23.535331 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.535323 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f2721af-325b-407d-8e1e-c1ed692291e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f2721af-325b-407d-8e1e-c1ed692291e8-istio-envoy\") pod \"0f2721af-325b-407d-8e1e-c1ed692291e8\" (UID: \"0f2721af-325b-407d-8e1e-c1ed692291e8\") " Oct 30 13:39:23.535527 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.535505 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2721af-325b-407d-8e1e-c1ed692291e8-credential-socket" pod "0f2721af-325b-407d-8e1e-c1ed692291e8" (UID: "0f2721af-325b-407d-8e1e-c1ed692291e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:23.535694 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.535666 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2721af-325b-407d-8e1e-c1ed692291e8-istio-data" pod "0f2721af-325b-407d-8e1e-c1ed692291e8" (UID: "0f2721af-325b-407d-8e1e-c1ed692291e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:23.536105 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.536080 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2721af-325b-407d-8e1e-c1ed692291e8-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:23.536208 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.536109 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f2721af-325b-407d-8e1e-c1ed692291e8-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:23.536409 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.536374 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2721af-325b-407d-8e1e-c1ed692291e8-workload-certs" pod "0f2721af-325b-407d-8e1e-c1ed692291e8" (UID: "0f2721af-325b-407d-8e1e-c1ed692291e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:23.536842 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.536804 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2721af-325b-407d-8e1e-c1ed692291e8-workload-socket" pod "0f2721af-325b-407d-8e1e-c1ed692291e8" (UID: "0f2721af-325b-407d-8e1e-c1ed692291e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:23.537176 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.537150 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f2721af-325b-407d-8e1e-c1ed692291e8-istiod-ca-cert" pod "0f2721af-325b-407d-8e1e-c1ed692291e8" (UID: "0f2721af-325b-407d-8e1e-c1ed692291e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:39:23.537989 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.537965 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f2721af-325b-407d-8e1e-c1ed692291e8-kube-api-access-xrl4f" pod "0f2721af-325b-407d-8e1e-c1ed692291e8" (UID: "0f2721af-325b-407d-8e1e-c1ed692291e8"). InnerVolumeSpecName "kube-api-access-xrl4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:23.537989 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.537973 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f2721af-325b-407d-8e1e-c1ed692291e8-istio-podinfo" pod "0f2721af-325b-407d-8e1e-c1ed692291e8" (UID: "0f2721af-325b-407d-8e1e-c1ed692291e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:39:23.538156 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.538134 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0b2c92-c82v5"] Oct 30 13:39:23.538514 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.538487 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f2721af-325b-407d-8e1e-c1ed692291e8-istio-token" pod "0f2721af-325b-407d-8e1e-c1ed692291e8" (UID: "0f2721af-325b-407d-8e1e-c1ed692291e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:23.538821 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.538802 4931 cpu_manager.go:414] "containerMap: removing container" podUID="67e16bff-a9c9-42d1-b787-dd6f0f98658d" containerName="revoke-keys" Oct 30 13:39:23.538892 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.538821 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67e16bff-a9c9-42d1-b787-dd6f0f98658d" containerName="revoke-keys" Oct 30 13:39:23.538892 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.538835 4931 cpu_manager.go:414] "containerMap: removing container" podUID="0f2721af-325b-407d-8e1e-c1ed692291e8" containerName="istio-proxy" Oct 30 13:39:23.538892 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.538840 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f2721af-325b-407d-8e1e-c1ed692291e8" containerName="istio-proxy" Oct 30 13:39:23.538892 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.538847 4931 cpu_manager.go:414] "containerMap: removing container" podUID="0f2721af-325b-407d-8e1e-c1ed692291e8" containerName="istio-proxy" Oct 30 13:39:23.538892 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.538852 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f2721af-325b-407d-8e1e-c1ed692291e8" containerName="istio-proxy" Oct 30 13:39:23.539157 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.538945 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="67e16bff-a9c9-42d1-b787-dd6f0f98658d" containerName="revoke-keys" Oct 30 13:39:23.539157 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.538959 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f2721af-325b-407d-8e1e-c1ed692291e8" containerName="istio-proxy" Oct 30 13:39:23.539157 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.538964 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f2721af-325b-407d-8e1e-c1ed692291e8" containerName="istio-proxy" Oct 30 13:39:23.540433 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.540410 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2721af-325b-407d-8e1e-c1ed692291e8-istio-envoy" pod "0f2721af-325b-407d-8e1e-c1ed692291e8" (UID: "0f2721af-325b-407d-8e1e-c1ed692291e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:23.542742 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.542723 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0b2c92-c82v5" Oct 30 13:39:23.563813 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.563790 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0b2c92-c82v5"] Oct 30 13:39:23.568586 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.568368 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:39:23.568586 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.568459 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:39:23.637646 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.637608 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8tbz\" (UniqueName: \"kubernetes.io/projected/17a57146-5cd3-432b-b96c-6d2f40ab81ee-kube-api-access-v8tbz\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-0b2c92-c82v5\" (UID: \"17a57146-5cd3-432b-b96c-6d2f40ab81ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0b2c92-c82v5" Oct 30 13:39:23.637834 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.637660 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17a57146-5cd3-432b-b96c-6d2f40ab81ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-0b2c92-c82v5\" (UID: \"17a57146-5cd3-432b-b96c-6d2f40ab81ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0b2c92-c82v5" Oct 30 13:39:23.637834 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.637806 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17a57146-5cd3-432b-b96c-6d2f40ab81ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-0b2c92-c82v5\" (UID: \"17a57146-5cd3-432b-b96c-6d2f40ab81ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0b2c92-c82v5" Oct 30 13:39:23.638068 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.638050 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrl4f\" (UniqueName: \"kubernetes.io/projected/0f2721af-325b-407d-8e1e-c1ed692291e8-kube-api-access-xrl4f\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:23.638147 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.638074 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f2721af-325b-407d-8e1e-c1ed692291e8-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:23.638147 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.638089 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f2721af-325b-407d-8e1e-c1ed692291e8-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:23.638147 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.638102 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f2721af-325b-407d-8e1e-c1ed692291e8-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:23.638147 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.638115 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f2721af-325b-407d-8e1e-c1ed692291e8-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:23.638147 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.638129 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2721af-325b-407d-8e1e-c1ed692291e8-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:23.638147 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.638142 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f2721af-325b-407d-8e1e-c1ed692291e8-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:23.738993 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.738960 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17a57146-5cd3-432b-b96c-6d2f40ab81ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-0b2c92-c82v5\" (UID: \"17a57146-5cd3-432b-b96c-6d2f40ab81ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0b2c92-c82v5" Oct 30 13:39:23.739168 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.739051 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17a57146-5cd3-432b-b96c-6d2f40ab81ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-0b2c92-c82v5\" (UID: \"17a57146-5cd3-432b-b96c-6d2f40ab81ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0b2c92-c82v5" Oct 30 13:39:23.739278 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.739247 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8tbz\" (UniqueName: \"kubernetes.io/projected/17a57146-5cd3-432b-b96c-6d2f40ab81ee-kube-api-access-v8tbz\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-0b2c92-c82v5\" (UID: \"17a57146-5cd3-432b-b96c-6d2f40ab81ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0b2c92-c82v5" Oct 30 13:39:23.739787 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.739764 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17a57146-5cd3-432b-b96c-6d2f40ab81ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-0b2c92-c82v5\" (UID: \"17a57146-5cd3-432b-b96c-6d2f40ab81ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0b2c92-c82v5" Oct 30 13:39:23.741380 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.741351 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17a57146-5cd3-432b-b96c-6d2f40ab81ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-0b2c92-c82v5\" (UID: \"17a57146-5cd3-432b-b96c-6d2f40ab81ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0b2c92-c82v5" Oct 30 13:39:23.763781 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.763758 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8tbz\" (UniqueName: \"kubernetes.io/projected/17a57146-5cd3-432b-b96c-6d2f40ab81ee-kube-api-access-v8tbz\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-0b2c92-c82v5\" (UID: \"17a57146-5cd3-432b-b96c-6d2f40ab81ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0b2c92-c82v5" Oct 30 13:39:23.875773 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:23.875700 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0b2c92-c82v5" Oct 30 13:39:24.185595 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:39:24.185551 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a57146_5cd3_432b_b96c_6d2f40ab81ee.slice/crio-14b406fae12619fd46879523802ca633545c248e8dab6084b2a9b6d48a029ce0 WatchSource:0}: Error finding container 14b406fae12619fd46879523802ca633545c248e8dab6084b2a9b6d48a029ce0: Status 404 returned error can't find the container with id 14b406fae12619fd46879523802ca633545c248e8dab6084b2a9b6d48a029ce0 Oct 30 13:39:24.186015 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:24.185992 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0b2c92-c82v5"] Oct 30 13:39:24.246760 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:24.246734 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0b2c92-c82v5" event={"ID":"17a57146-5cd3-432b-b96c-6d2f40ab81ee","Type":"ContainerStarted","Data":"14b406fae12619fd46879523802ca633545c248e8dab6084b2a9b6d48a029ce0"} Oct 30 13:39:24.248231 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:24.248209 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" Oct 30 13:39:24.248356 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:24.248202 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx" event={"ID":"0f2721af-325b-407d-8e1e-c1ed692291e8","Type":"ContainerDied","Data":"d15af90e1aa68bedfe7f54716efc3cb2cc8253ad41e0b32d8c62afac3bb51d15"} Oct 30 13:39:24.248408 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:24.248358 4931 scope.go:122] "RemoveContainer" containerID="21e09ea257786f910869b4a2138a4b0d8977f6fc51a882f77c64709d4224f27b" Oct 30 13:39:24.312537 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:24.312505 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx"] Oct 30 13:39:24.325453 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:24.325430 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx"] Oct 30 13:39:25.258927 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:25.258896 4931 generic.go:356] "Generic (PLEG): container finished" podID="db26d540-4441-49aa-be9f-51edab84b99c" containerID="5818472411fbdb6fe85ea3df7c0044ccb9f4f25d61bf7fe97baec779d9a1e93a" exitCode=0 Oct 30 13:39:25.259296 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:25.258966 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" event={"ID":"db26d540-4441-49aa-be9f-51edab84b99c","Type":"ContainerDied","Data":"5818472411fbdb6fe85ea3df7c0044ccb9f4f25d61bf7fe97baec779d9a1e93a"} Oct 30 13:39:25.259296 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:25.259046 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" event={"ID":"db26d540-4441-49aa-be9f-51edab84b99c","Type":"ContainerDied","Data":"0d792b7f239d80c08be48da062f9bf6b0b90c76986f383a65275337fa418593f"} Oct 30 13:39:25.259296 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:25.259068 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d792b7f239d80c08be48da062f9bf6b0b90c76986f383a65275337fa418593f" Oct 30 13:39:25.260745 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:25.260719 4931 generic.go:356] "Generic (PLEG): container finished" podID="17a57146-5cd3-432b-b96c-6d2f40ab81ee" containerID="da3fe135651e4337b424c89363e9515865ad08e09a725a678d68080d32228d95" exitCode=0 Oct 30 13:39:25.260850 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:25.260796 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0b2c92-c82v5" event={"ID":"17a57146-5cd3-432b-b96c-6d2f40ab81ee","Type":"ContainerDied","Data":"da3fe135651e4337b424c89363e9515865ad08e09a725a678d68080d32228d95"} Oct 30 13:39:25.273783 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:25.273759 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" Oct 30 13:39:25.359972 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:25.359935 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db26d540-4441-49aa-be9f-51edab84b99c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db26d540-4441-49aa-be9f-51edab84b99c-credential-socket\") pod \"db26d540-4441-49aa-be9f-51edab84b99c\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " Oct 30 13:39:25.360131 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:25.360085 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db26d540-4441-49aa-be9f-51edab84b99c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db26d540-4441-49aa-be9f-51edab84b99c-istiod-ca-cert\") pod \"db26d540-4441-49aa-be9f-51edab84b99c\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " Oct 30 13:39:25.360236 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:25.360219 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db26d540-4441-49aa-be9f-51edab84b99c-istio-token\" (UniqueName: \"kubernetes.io/projected/db26d540-4441-49aa-be9f-51edab84b99c-istio-token\") pod \"db26d540-4441-49aa-be9f-51edab84b99c\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " Oct 30 13:39:25.360354 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:25.360312 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db26d540-4441-49aa-be9f-51edab84b99c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db26d540-4441-49aa-be9f-51edab84b99c-istio-envoy\") pod \"db26d540-4441-49aa-be9f-51edab84b99c\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " Oct 30 13:39:25.360428 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:25.360352 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db26d540-4441-49aa-be9f-51edab84b99c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db26d540-4441-49aa-be9f-51edab84b99c-istio-podinfo\") pod \"db26d540-4441-49aa-be9f-51edab84b99c\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " Oct 30 13:39:25.360428 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:25.360396 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db26d540-4441-49aa-be9f-51edab84b99c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db26d540-4441-49aa-be9f-51edab84b99c-istio-data\") pod \"db26d540-4441-49aa-be9f-51edab84b99c\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " Oct 30 13:39:25.360547 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:25.360439 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db26d540-4441-49aa-be9f-51edab84b99c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db26d540-4441-49aa-be9f-51edab84b99c-workload-certs\") pod \"db26d540-4441-49aa-be9f-51edab84b99c\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " Oct 30 13:39:25.360547 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:25.360468 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db26d540-4441-49aa-be9f-51edab84b99c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db26d540-4441-49aa-be9f-51edab84b99c-workload-socket\") pod \"db26d540-4441-49aa-be9f-51edab84b99c\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " Oct 30 13:39:25.360547 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:25.360511 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db26d540-4441-49aa-be9f-51edab84b99c-kube-api-access-4dkhw\" (UniqueName: \"kubernetes.io/projected/db26d540-4441-49aa-be9f-51edab84b99c-kube-api-access-4dkhw\") pod \"db26d540-4441-49aa-be9f-51edab84b99c\" (UID: \"db26d540-4441-49aa-be9f-51edab84b99c\") " Oct 30 13:39:25.363240 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:25.361582 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db26d540-4441-49aa-be9f-51edab84b99c-workload-certs" pod "db26d540-4441-49aa-be9f-51edab84b99c" (UID: "db26d540-4441-49aa-be9f-51edab84b99c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:25.363240 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:25.361597 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db26d540-4441-49aa-be9f-51edab84b99c-credential-socket" pod "db26d540-4441-49aa-be9f-51edab84b99c" (UID: "db26d540-4441-49aa-be9f-51edab84b99c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:25.363240 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:25.361665 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db26d540-4441-49aa-be9f-51edab84b99c-istiod-ca-cert" pod "db26d540-4441-49aa-be9f-51edab84b99c" (UID: "db26d540-4441-49aa-be9f-51edab84b99c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:39:25.363240 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:25.361844 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db26d540-4441-49aa-be9f-51edab84b99c-workload-socket" pod "db26d540-4441-49aa-be9f-51edab84b99c" (UID: "db26d540-4441-49aa-be9f-51edab84b99c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:25.363240 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:25.362597 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db26d540-4441-49aa-be9f-51edab84b99c-istio-data" pod "db26d540-4441-49aa-be9f-51edab84b99c" (UID: "db26d540-4441-49aa-be9f-51edab84b99c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:25.364337 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:25.364312 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db26d540-4441-49aa-be9f-51edab84b99c-istio-podinfo" pod "db26d540-4441-49aa-be9f-51edab84b99c" (UID: "db26d540-4441-49aa-be9f-51edab84b99c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:39:25.364467 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:25.364386 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db26d540-4441-49aa-be9f-51edab84b99c-istio-envoy" pod "db26d540-4441-49aa-be9f-51edab84b99c" (UID: "db26d540-4441-49aa-be9f-51edab84b99c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:25.365397 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:25.365357 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db26d540-4441-49aa-be9f-51edab84b99c-istio-token" pod "db26d540-4441-49aa-be9f-51edab84b99c" (UID: "db26d540-4441-49aa-be9f-51edab84b99c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:25.370453 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:25.370428 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db26d540-4441-49aa-be9f-51edab84b99c-kube-api-access-4dkhw" pod "db26d540-4441-49aa-be9f-51edab84b99c" (UID: "db26d540-4441-49aa-be9f-51edab84b99c"). InnerVolumeSpecName "kube-api-access-4dkhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:25.462037 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:25.462009 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db26d540-4441-49aa-be9f-51edab84b99c-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:25.462037 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:25.462035 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db26d540-4441-49aa-be9f-51edab84b99c-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:25.462238 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:25.462045 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dkhw\" (UniqueName: \"kubernetes.io/projected/db26d540-4441-49aa-be9f-51edab84b99c-kube-api-access-4dkhw\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:25.462238 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:25.462055 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db26d540-4441-49aa-be9f-51edab84b99c-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:25.462238 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:25.462064 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db26d540-4441-49aa-be9f-51edab84b99c-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:25.462238 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:25.462073 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db26d540-4441-49aa-be9f-51edab84b99c-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:25.462238 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:25.462081 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db26d540-4441-49aa-be9f-51edab84b99c-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:25.462238 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:25.462089 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db26d540-4441-49aa-be9f-51edab84b99c-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:25.462238 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:25.462097 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db26d540-4441-49aa-be9f-51edab84b99c-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:25.514829 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:25.514754 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f2721af-325b-407d-8e1e-c1ed692291e8" path="/var/lib/kubelet/pods/0f2721af-325b-407d-8e1e-c1ed692291e8/volumes" Oct 30 13:39:26.267200 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:26.267165 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw" Oct 30 13:39:26.349303 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:26.349258 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw"] Oct 30 13:39:26.361466 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:26.361440 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw"] Oct 30 13:39:26.436199 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:26.436172 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0b2c92-c82v5" Oct 30 13:39:26.474035 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:26.474008 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17a57146-5cd3-432b-b96c-6d2f40ab81ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17a57146-5cd3-432b-b96c-6d2f40ab81ee-maas-api-xks-ca\") pod \"17a57146-5cd3-432b-b96c-6d2f40ab81ee\" (UID: \"17a57146-5cd3-432b-b96c-6d2f40ab81ee\") " Oct 30 13:39:26.474171 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:26.474050 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17a57146-5cd3-432b-b96c-6d2f40ab81ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17a57146-5cd3-432b-b96c-6d2f40ab81ee-maas-api-openshift-ca\") pod \"17a57146-5cd3-432b-b96c-6d2f40ab81ee\" (UID: \"17a57146-5cd3-432b-b96c-6d2f40ab81ee\") " Oct 30 13:39:26.474171 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:26.474161 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17a57146-5cd3-432b-b96c-6d2f40ab81ee-kube-api-access-v8tbz\" (UniqueName: \"kubernetes.io/projected/17a57146-5cd3-432b-b96c-6d2f40ab81ee-kube-api-access-v8tbz\") pod \"17a57146-5cd3-432b-b96c-6d2f40ab81ee\" (UID: \"17a57146-5cd3-432b-b96c-6d2f40ab81ee\") " Oct 30 13:39:26.474484 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:26.474458 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a57146-5cd3-432b-b96c-6d2f40ab81ee-maas-api-openshift-ca" pod "17a57146-5cd3-432b-b96c-6d2f40ab81ee" (UID: "17a57146-5cd3-432b-b96c-6d2f40ab81ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:39:26.474745 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:26.474727 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17a57146-5cd3-432b-b96c-6d2f40ab81ee-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:26.476104 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:26.476075 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17a57146-5cd3-432b-b96c-6d2f40ab81ee-maas-api-xks-ca" pod "17a57146-5cd3-432b-b96c-6d2f40ab81ee" (UID: "17a57146-5cd3-432b-b96c-6d2f40ab81ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:39:26.476318 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:26.476254 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a57146-5cd3-432b-b96c-6d2f40ab81ee-kube-api-access-v8tbz" pod "17a57146-5cd3-432b-b96c-6d2f40ab81ee" (UID: "17a57146-5cd3-432b-b96c-6d2f40ab81ee"). InnerVolumeSpecName "kube-api-access-v8tbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:26.575750 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:26.575667 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17a57146-5cd3-432b-b96c-6d2f40ab81ee-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:26.575750 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:26.575693 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8tbz\" (UniqueName: \"kubernetes.io/projected/17a57146-5cd3-432b-b96c-6d2f40ab81ee-kube-api-access-v8tbz\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:27.276849 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:27.276812 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0b2c92-c82v5" event={"ID":"17a57146-5cd3-432b-b96c-6d2f40ab81ee","Type":"ContainerDied","Data":"14b406fae12619fd46879523802ca633545c248e8dab6084b2a9b6d48a029ce0"} Oct 30 13:39:27.277261 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:27.276870 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14b406fae12619fd46879523802ca633545c248e8dab6084b2a9b6d48a029ce0" Oct 30 13:39:27.277261 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:27.276827 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0b2c92-c82v5" Oct 30 13:39:27.513797 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:27.513757 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db26d540-4441-49aa-be9f-51edab84b99c" path="/var/lib/kubelet/pods/db26d540-4441-49aa-be9f-51edab84b99c/volumes" Oct 30 13:39:27.695158 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:27.695120 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-bcd76c-5qgjz"] Oct 30 13:39:27.696011 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:27.695986 4931 cpu_manager.go:414] "containerMap: removing container" podUID="db26d540-4441-49aa-be9f-51edab84b99c" containerName="istio-proxy" Oct 30 13:39:27.696126 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:27.696016 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db26d540-4441-49aa-be9f-51edab84b99c" containerName="istio-proxy" Oct 30 13:39:27.696126 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:27.696052 4931 cpu_manager.go:414] "containerMap: removing container" podUID="17a57146-5cd3-432b-b96c-6d2f40ab81ee" containerName="revoke-keys" Oct 30 13:39:27.696126 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:27.696062 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17a57146-5cd3-432b-b96c-6d2f40ab81ee" containerName="revoke-keys" Oct 30 13:39:27.696351 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:27.696226 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="db26d540-4441-49aa-be9f-51edab84b99c" containerName="istio-proxy" Oct 30 13:39:27.696351 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:27.696247 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="17a57146-5cd3-432b-b96c-6d2f40ab81ee" containerName="revoke-keys" Oct 30 13:39:27.701862 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:27.701841 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-bcd76c-5qgjz" Oct 30 13:39:27.722250 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:27.722214 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-bcd76c-5qgjz"] Oct 30 13:39:27.722578 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:27.722557 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:39:27.723192 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:27.723170 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:39:27.817078 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:27.817043 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lnhh\" (UniqueName: \"kubernetes.io/projected/ea8ce722-998e-4d37-8e05-8e659833303e-kube-api-access-4lnhh\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-bcd76c-5qgjz\" (UID: \"ea8ce722-998e-4d37-8e05-8e659833303e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-bcd76c-5qgjz" Oct 30 13:39:27.817314 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:27.817121 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea8ce722-998e-4d37-8e05-8e659833303e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-bcd76c-5qgjz\" (UID: \"ea8ce722-998e-4d37-8e05-8e659833303e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-bcd76c-5qgjz" Oct 30 13:39:27.817409 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:27.817338 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea8ce722-998e-4d37-8e05-8e659833303e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-bcd76c-5qgjz\" (UID: \"ea8ce722-998e-4d37-8e05-8e659833303e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-bcd76c-5qgjz" Oct 30 13:39:27.918343 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:27.918304 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea8ce722-998e-4d37-8e05-8e659833303e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-bcd76c-5qgjz\" (UID: \"ea8ce722-998e-4d37-8e05-8e659833303e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-bcd76c-5qgjz" Oct 30 13:39:27.918518 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:27.918396 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lnhh\" (UniqueName: \"kubernetes.io/projected/ea8ce722-998e-4d37-8e05-8e659833303e-kube-api-access-4lnhh\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-bcd76c-5qgjz\" (UID: \"ea8ce722-998e-4d37-8e05-8e659833303e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-bcd76c-5qgjz" Oct 30 13:39:27.918518 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:27.918455 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea8ce722-998e-4d37-8e05-8e659833303e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-bcd76c-5qgjz\" (UID: \"ea8ce722-998e-4d37-8e05-8e659833303e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-bcd76c-5qgjz" Oct 30 13:39:27.918996 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:27.918967 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea8ce722-998e-4d37-8e05-8e659833303e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-bcd76c-5qgjz\" (UID: \"ea8ce722-998e-4d37-8e05-8e659833303e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-bcd76c-5qgjz" Oct 30 13:39:27.920839 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:27.920815 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea8ce722-998e-4d37-8e05-8e659833303e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-bcd76c-5qgjz\" (UID: \"ea8ce722-998e-4d37-8e05-8e659833303e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-bcd76c-5qgjz" Oct 30 13:39:27.943632 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:27.943596 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lnhh\" (UniqueName: \"kubernetes.io/projected/ea8ce722-998e-4d37-8e05-8e659833303e-kube-api-access-4lnhh\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-bcd76c-5qgjz\" (UID: \"ea8ce722-998e-4d37-8e05-8e659833303e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-bcd76c-5qgjz" Oct 30 13:39:28.027313 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:28.027227 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-bcd76c-5qgjz" Oct 30 13:39:28.343635 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:28.343608 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-bcd76c-5qgjz"] Oct 30 13:39:28.344409 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:39:28.344385 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea8ce722_998e_4d37_8e05_8e659833303e.slice/crio-28d10af78ae75e87d9a13defa5a2f9085de005d2dca258c76b24e2aa1f17fcaf WatchSource:0}: Error finding container 28d10af78ae75e87d9a13defa5a2f9085de005d2dca258c76b24e2aa1f17fcaf: Status 404 returned error can't find the container with id 28d10af78ae75e87d9a13defa5a2f9085de005d2dca258c76b24e2aa1f17fcaf Oct 30 13:39:29.293229 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:29.293194 4931 generic.go:356] "Generic (PLEG): container finished" podID="ea8ce722-998e-4d37-8e05-8e659833303e" containerID="6c8d4ffce367e5f54868df833edbf1159e555ee9cd00cdf6c1fe87da2d29c71f" exitCode=0 Oct 30 13:39:29.293978 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:29.293857 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-bcd76c-5qgjz" event={"ID":"ea8ce722-998e-4d37-8e05-8e659833303e","Type":"ContainerDied","Data":"6c8d4ffce367e5f54868df833edbf1159e555ee9cd00cdf6c1fe87da2d29c71f"} Oct 30 13:39:29.294227 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:29.294173 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-bcd76c-5qgjz" event={"ID":"ea8ce722-998e-4d37-8e05-8e659833303e","Type":"ContainerStarted","Data":"28d10af78ae75e87d9a13defa5a2f9085de005d2dca258c76b24e2aa1f17fcaf"} Oct 30 13:39:29.871470 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:29.871434 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr"] Oct 30 13:39:29.871890 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:29.871659 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" podUID="24214a0a-42ce-4fa2-b4ef-9a28771278ce" containerName="istio-proxy" containerID="cri-o://96f1aa904a20156b00cbd3a7794a9a350fe787d6bc80c53a90a677ca350f649a" gracePeriod=30 Oct 30 13:39:30.472576 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:30.472547 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-bcd76c-5qgjz" Oct 30 13:39:30.579711 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:30.579633 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea8ce722-998e-4d37-8e05-8e659833303e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea8ce722-998e-4d37-8e05-8e659833303e-maas-api-openshift-ca\") pod \"ea8ce722-998e-4d37-8e05-8e659833303e\" (UID: \"ea8ce722-998e-4d37-8e05-8e659833303e\") " Oct 30 13:39:30.579888 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:30.579750 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea8ce722-998e-4d37-8e05-8e659833303e-kube-api-access-4lnhh\" (UniqueName: \"kubernetes.io/projected/ea8ce722-998e-4d37-8e05-8e659833303e-kube-api-access-4lnhh\") pod \"ea8ce722-998e-4d37-8e05-8e659833303e\" (UID: \"ea8ce722-998e-4d37-8e05-8e659833303e\") " Oct 30 13:39:30.579888 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:30.579867 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea8ce722-998e-4d37-8e05-8e659833303e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea8ce722-998e-4d37-8e05-8e659833303e-maas-api-xks-ca\") pod \"ea8ce722-998e-4d37-8e05-8e659833303e\" (UID: \"ea8ce722-998e-4d37-8e05-8e659833303e\") " Oct 30 13:39:30.580060 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:30.580040 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea8ce722-998e-4d37-8e05-8e659833303e-maas-api-openshift-ca" pod "ea8ce722-998e-4d37-8e05-8e659833303e" (UID: "ea8ce722-998e-4d37-8e05-8e659833303e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:39:30.580717 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:30.580694 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea8ce722-998e-4d37-8e05-8e659833303e-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:30.582191 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:30.582160 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea8ce722-998e-4d37-8e05-8e659833303e-kube-api-access-4lnhh" pod "ea8ce722-998e-4d37-8e05-8e659833303e" (UID: "ea8ce722-998e-4d37-8e05-8e659833303e"). InnerVolumeSpecName "kube-api-access-4lnhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:30.582605 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:30.582579 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea8ce722-998e-4d37-8e05-8e659833303e-maas-api-xks-ca" pod "ea8ce722-998e-4d37-8e05-8e659833303e" (UID: "ea8ce722-998e-4d37-8e05-8e659833303e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:39:30.655451 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:30.655420 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:39:30.682043 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:30.682012 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea8ce722-998e-4d37-8e05-8e659833303e-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:30.682043 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:30.682039 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lnhh\" (UniqueName: \"kubernetes.io/projected/ea8ce722-998e-4d37-8e05-8e659833303e-kube-api-access-4lnhh\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:31.311233 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:31.311201 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-bcd76c-5qgjz" event={"ID":"ea8ce722-998e-4d37-8e05-8e659833303e","Type":"ContainerDied","Data":"28d10af78ae75e87d9a13defa5a2f9085de005d2dca258c76b24e2aa1f17fcaf"} Oct 30 13:39:31.311680 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:31.311236 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-bcd76c-5qgjz" Oct 30 13:39:31.311680 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:31.311249 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28d10af78ae75e87d9a13defa5a2f9085de005d2dca258c76b24e2aa1f17fcaf" Oct 30 13:39:35.155384 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.154972 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" Oct 30 13:39:35.243939 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.243889 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24214a0a-42ce-4fa2-b4ef-9a28771278ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24214a0a-42ce-4fa2-b4ef-9a28771278ce-credential-socket\") pod \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " Oct 30 13:39:35.244135 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.243982 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24214a0a-42ce-4fa2-b4ef-9a28771278ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24214a0a-42ce-4fa2-b4ef-9a28771278ce-istio-envoy\") pod \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " Oct 30 13:39:35.244135 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.244041 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24214a0a-42ce-4fa2-b4ef-9a28771278ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24214a0a-42ce-4fa2-b4ef-9a28771278ce-workload-certs\") pod \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " Oct 30 13:39:35.244135 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.244098 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24214a0a-42ce-4fa2-b4ef-9a28771278ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24214a0a-42ce-4fa2-b4ef-9a28771278ce-istio-data\") pod \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " Oct 30 13:39:35.244135 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.244128 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24214a0a-42ce-4fa2-b4ef-9a28771278ce-kube-api-access-tdc9h\" (UniqueName: \"kubernetes.io/projected/24214a0a-42ce-4fa2-b4ef-9a28771278ce-kube-api-access-tdc9h\") pod \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " Oct 30 13:39:35.244419 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.244187 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24214a0a-42ce-4fa2-b4ef-9a28771278ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24214a0a-42ce-4fa2-b4ef-9a28771278ce-workload-socket\") pod \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " Oct 30 13:39:35.244419 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.244221 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24214a0a-42ce-4fa2-b4ef-9a28771278ce-istio-token\" (UniqueName: \"kubernetes.io/projected/24214a0a-42ce-4fa2-b4ef-9a28771278ce-istio-token\") pod \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " Oct 30 13:39:35.244419 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.244222 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24214a0a-42ce-4fa2-b4ef-9a28771278ce-credential-socket" pod "24214a0a-42ce-4fa2-b4ef-9a28771278ce" (UID: "24214a0a-42ce-4fa2-b4ef-9a28771278ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:35.244419 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.244298 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24214a0a-42ce-4fa2-b4ef-9a28771278ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24214a0a-42ce-4fa2-b4ef-9a28771278ce-istiod-ca-cert\") pod \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " Oct 30 13:39:35.244419 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.244375 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24214a0a-42ce-4fa2-b4ef-9a28771278ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24214a0a-42ce-4fa2-b4ef-9a28771278ce-istio-podinfo\") pod \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\" (UID: \"24214a0a-42ce-4fa2-b4ef-9a28771278ce\") " Oct 30 13:39:35.244701 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.244439 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24214a0a-42ce-4fa2-b4ef-9a28771278ce-workload-certs" pod "24214a0a-42ce-4fa2-b4ef-9a28771278ce" (UID: "24214a0a-42ce-4fa2-b4ef-9a28771278ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:35.244766 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.244735 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24214a0a-42ce-4fa2-b4ef-9a28771278ce-workload-socket" pod "24214a0a-42ce-4fa2-b4ef-9a28771278ce" (UID: "24214a0a-42ce-4fa2-b4ef-9a28771278ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:35.244766 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.244754 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24214a0a-42ce-4fa2-b4ef-9a28771278ce-istio-data" pod "24214a0a-42ce-4fa2-b4ef-9a28771278ce" (UID: "24214a0a-42ce-4fa2-b4ef-9a28771278ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:35.244858 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.244790 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24214a0a-42ce-4fa2-b4ef-9a28771278ce-istiod-ca-cert" pod "24214a0a-42ce-4fa2-b4ef-9a28771278ce" (UID: "24214a0a-42ce-4fa2-b4ef-9a28771278ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:39:35.245039 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.245013 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24214a0a-42ce-4fa2-b4ef-9a28771278ce-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:35.245101 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.245042 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24214a0a-42ce-4fa2-b4ef-9a28771278ce-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:35.245101 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.245056 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24214a0a-42ce-4fa2-b4ef-9a28771278ce-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:35.245101 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.245069 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24214a0a-42ce-4fa2-b4ef-9a28771278ce-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:35.245101 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.245084 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24214a0a-42ce-4fa2-b4ef-9a28771278ce-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:35.248865 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.248766 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24214a0a-42ce-4fa2-b4ef-9a28771278ce-kube-api-access-tdc9h" pod "24214a0a-42ce-4fa2-b4ef-9a28771278ce" (UID: "24214a0a-42ce-4fa2-b4ef-9a28771278ce"). InnerVolumeSpecName "kube-api-access-tdc9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:35.248865 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.248810 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24214a0a-42ce-4fa2-b4ef-9a28771278ce-istio-token" pod "24214a0a-42ce-4fa2-b4ef-9a28771278ce" (UID: "24214a0a-42ce-4fa2-b4ef-9a28771278ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:35.248865 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.248830 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24214a0a-42ce-4fa2-b4ef-9a28771278ce-istio-envoy" pod "24214a0a-42ce-4fa2-b4ef-9a28771278ce" (UID: "24214a0a-42ce-4fa2-b4ef-9a28771278ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:35.249900 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.249873 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24214a0a-42ce-4fa2-b4ef-9a28771278ce-istio-podinfo" pod "24214a0a-42ce-4fa2-b4ef-9a28771278ce" (UID: "24214a0a-42ce-4fa2-b4ef-9a28771278ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:39:35.346322 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.346255 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24214a0a-42ce-4fa2-b4ef-9a28771278ce-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:35.346322 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.346317 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24214a0a-42ce-4fa2-b4ef-9a28771278ce-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:35.346322 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.346328 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdc9h\" (UniqueName: \"kubernetes.io/projected/24214a0a-42ce-4fa2-b4ef-9a28771278ce-kube-api-access-tdc9h\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:35.346556 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.346337 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24214a0a-42ce-4fa2-b4ef-9a28771278ce-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:35.375373 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.375327 4931 generic.go:356] "Generic (PLEG): container finished" podID="24214a0a-42ce-4fa2-b4ef-9a28771278ce" containerID="96f1aa904a20156b00cbd3a7794a9a350fe787d6bc80c53a90a677ca350f649a" exitCode=0 Oct 30 13:39:35.375598 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.375391 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" Oct 30 13:39:35.375757 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.375413 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" event={"ID":"24214a0a-42ce-4fa2-b4ef-9a28771278ce","Type":"ContainerDied","Data":"96f1aa904a20156b00cbd3a7794a9a350fe787d6bc80c53a90a677ca350f649a"} Oct 30 13:39:35.375831 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.375757 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr" event={"ID":"24214a0a-42ce-4fa2-b4ef-9a28771278ce","Type":"ContainerDied","Data":"c44b921de9f6301cf69137c24f2eeee69e78e6b29fdf73358fc558c7e0777cfb"} Oct 30 13:39:35.375831 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.375802 4931 scope.go:122] "RemoveContainer" containerID="96f1aa904a20156b00cbd3a7794a9a350fe787d6bc80c53a90a677ca350f649a" Oct 30 13:39:35.391148 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.391120 4931 scope.go:122] "RemoveContainer" containerID="96f1aa904a20156b00cbd3a7794a9a350fe787d6bc80c53a90a677ca350f649a" Oct 30 13:39:35.391564 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:39:35.391541 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96f1aa904a20156b00cbd3a7794a9a350fe787d6bc80c53a90a677ca350f649a\": container with ID starting with 96f1aa904a20156b00cbd3a7794a9a350fe787d6bc80c53a90a677ca350f649a not found: ID does not exist" containerID="96f1aa904a20156b00cbd3a7794a9a350fe787d6bc80c53a90a677ca350f649a" Oct 30 13:39:35.391665 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.391617 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96f1aa904a20156b00cbd3a7794a9a350fe787d6bc80c53a90a677ca350f649a"} err="failed to get container status \"96f1aa904a20156b00cbd3a7794a9a350fe787d6bc80c53a90a677ca350f649a\": rpc error: code = NotFound desc = could not find container \"96f1aa904a20156b00cbd3a7794a9a350fe787d6bc80c53a90a677ca350f649a\": container with ID starting with 96f1aa904a20156b00cbd3a7794a9a350fe787d6bc80c53a90a677ca350f649a not found: ID does not exist" Oct 30 13:39:35.439972 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.439914 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr"] Oct 30 13:39:35.443236 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.443213 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr"] Oct 30 13:39:35.562755 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:35.562693 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24214a0a-42ce-4fa2-b4ef-9a28771278ce" path="/var/lib/kubelet/pods/24214a0a-42ce-4fa2-b4ef-9a28771278ce/volumes" Oct 30 13:39:37.036028 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:37.035990 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws"] Oct 30 13:39:37.036478 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:37.036217 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" podUID="e46deba0-1ef3-4b54-9821-d4fbf394c9a6" containerName="istio-proxy" containerID="cri-o://bdd6ac7ab3e0c1e2e426a3c54fe195f84b2675d04e68c551dd1ea7314ca3c911" gracePeriod=30 Oct 30 13:39:37.352089 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:39:37.352007 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:39:42.316449 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.316388 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" Oct 30 13:39:42.350974 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.350940 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-istio-token\" (UniqueName: \"kubernetes.io/projected/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-istio-token\") pod \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " Oct 30 13:39:42.350974 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.350977 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-istio-envoy\") pod \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " Oct 30 13:39:42.351218 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.350995 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-istio-data\") pod \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " Oct 30 13:39:42.351218 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.351018 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-workload-socket\") pod \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " Oct 30 13:39:42.351218 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.351052 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-credential-socket\") pod \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " Oct 30 13:39:42.351218 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.351125 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-istio-podinfo\") pod \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " Oct 30 13:39:42.351218 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.351176 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-kube-api-access-8qbvq\" (UniqueName: \"kubernetes.io/projected/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-kube-api-access-8qbvq\") pod \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " Oct 30 13:39:42.351555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.351249 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-workload-certs\") pod \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " Oct 30 13:39:42.351555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.351304 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-istiod-ca-cert\") pod \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\" (UID: \"e46deba0-1ef3-4b54-9821-d4fbf394c9a6\") " Oct 30 13:39:42.351555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.351357 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-istio-data" pod "e46deba0-1ef3-4b54-9821-d4fbf394c9a6" (UID: "e46deba0-1ef3-4b54-9821-d4fbf394c9a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:42.351555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.351414 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-workload-socket" pod "e46deba0-1ef3-4b54-9821-d4fbf394c9a6" (UID: "e46deba0-1ef3-4b54-9821-d4fbf394c9a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:42.351555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.351431 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-credential-socket" pod "e46deba0-1ef3-4b54-9821-d4fbf394c9a6" (UID: "e46deba0-1ef3-4b54-9821-d4fbf394c9a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:42.351810 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.351591 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-workload-certs" pod "e46deba0-1ef3-4b54-9821-d4fbf394c9a6" (UID: "e46deba0-1ef3-4b54-9821-d4fbf394c9a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:42.351865 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.351806 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-istiod-ca-cert" pod "e46deba0-1ef3-4b54-9821-d4fbf394c9a6" (UID: "e46deba0-1ef3-4b54-9821-d4fbf394c9a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:39:42.353048 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.352066 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:42.353048 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.352091 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:42.353048 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.352105 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:42.353048 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.352119 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:42.353048 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.352155 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:42.353538 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.353514 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-istio-envoy" pod "e46deba0-1ef3-4b54-9821-d4fbf394c9a6" (UID: "e46deba0-1ef3-4b54-9821-d4fbf394c9a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:42.353668 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.353646 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-kube-api-access-8qbvq" pod "e46deba0-1ef3-4b54-9821-d4fbf394c9a6" (UID: "e46deba0-1ef3-4b54-9821-d4fbf394c9a6"). InnerVolumeSpecName "kube-api-access-8qbvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:42.353716 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.353701 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-istio-token" pod "e46deba0-1ef3-4b54-9821-d4fbf394c9a6" (UID: "e46deba0-1ef3-4b54-9821-d4fbf394c9a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:42.353891 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.353874 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-istio-podinfo" pod "e46deba0-1ef3-4b54-9821-d4fbf394c9a6" (UID: "e46deba0-1ef3-4b54-9821-d4fbf394c9a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:39:42.429343 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.429306 4931 generic.go:356] "Generic (PLEG): container finished" podID="e46deba0-1ef3-4b54-9821-d4fbf394c9a6" containerID="bdd6ac7ab3e0c1e2e426a3c54fe195f84b2675d04e68c551dd1ea7314ca3c911" exitCode=0 Oct 30 13:39:42.429541 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.429379 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" Oct 30 13:39:42.429541 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.429397 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" event={"ID":"e46deba0-1ef3-4b54-9821-d4fbf394c9a6","Type":"ContainerDied","Data":"bdd6ac7ab3e0c1e2e426a3c54fe195f84b2675d04e68c551dd1ea7314ca3c911"} Oct 30 13:39:42.429541 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.429465 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws" event={"ID":"e46deba0-1ef3-4b54-9821-d4fbf394c9a6","Type":"ContainerDied","Data":"4dbfb122067215b62beecc0e17c763d6c4fe586ad87830e936a1e21a7421127b"} Oct 30 13:39:42.429541 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.429493 4931 scope.go:122] "RemoveContainer" containerID="bdd6ac7ab3e0c1e2e426a3c54fe195f84b2675d04e68c551dd1ea7314ca3c911" Oct 30 13:39:42.441238 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.441216 4931 scope.go:122] "RemoveContainer" containerID="bdd6ac7ab3e0c1e2e426a3c54fe195f84b2675d04e68c551dd1ea7314ca3c911" Oct 30 13:39:42.441547 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:39:42.441525 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdd6ac7ab3e0c1e2e426a3c54fe195f84b2675d04e68c551dd1ea7314ca3c911\": container with ID starting with bdd6ac7ab3e0c1e2e426a3c54fe195f84b2675d04e68c551dd1ea7314ca3c911 not found: ID does not exist" containerID="bdd6ac7ab3e0c1e2e426a3c54fe195f84b2675d04e68c551dd1ea7314ca3c911" Oct 30 13:39:42.441596 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.441556 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdd6ac7ab3e0c1e2e426a3c54fe195f84b2675d04e68c551dd1ea7314ca3c911"} err="failed to get container status \"bdd6ac7ab3e0c1e2e426a3c54fe195f84b2675d04e68c551dd1ea7314ca3c911\": rpc error: code = NotFound desc = could not find container \"bdd6ac7ab3e0c1e2e426a3c54fe195f84b2675d04e68c551dd1ea7314ca3c911\": container with ID starting with bdd6ac7ab3e0c1e2e426a3c54fe195f84b2675d04e68c551dd1ea7314ca3c911 not found: ID does not exist" Oct 30 13:39:42.453208 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.453186 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:42.453208 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.453209 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:42.453377 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.453218 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:42.453377 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.453227 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qbvq\" (UniqueName: \"kubernetes.io/projected/e46deba0-1ef3-4b54-9821-d4fbf394c9a6-kube-api-access-8qbvq\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:42.490019 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.489979 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws"] Oct 30 13:39:42.495187 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:42.495163 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws"] Oct 30 13:39:43.514310 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:43.514203 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e46deba0-1ef3-4b54-9821-d4fbf394c9a6" path="/var/lib/kubelet/pods/e46deba0-1ef3-4b54-9821-d4fbf394c9a6/volumes" Oct 30 13:39:49.240886 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:49.240848 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0b2c92-c82v5"] Oct 30 13:39:49.253581 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:49.253542 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0b2c92-c82v5"] Oct 30 13:39:49.513323 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:49.513239 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17a57146-5cd3-432b-b96c-6d2f40ab81ee" path="/var/lib/kubelet/pods/17a57146-5cd3-432b-b96c-6d2f40ab81ee/volumes" Oct 30 13:39:49.953551 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:49.953506 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e9d641-rhwl7"] Oct 30 13:39:49.958567 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:49.958532 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e9d641-rhwl7"] Oct 30 13:39:51.513877 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:51.513839 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67e16bff-a9c9-42d1-b787-dd6f0f98658d" path="/var/lib/kubelet/pods/67e16bff-a9c9-42d1-b787-dd6f0f98658d/volumes" Oct 30 13:39:53.261719 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:53.261686 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d"] Oct 30 13:39:53.262285 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:53.261962 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d" podUID="80f8171d-05ef-44b2-af37-12ea086b0334" containerName="maas-api" containerID="cri-o://797ebd0f0981f347f9c69624b9368825622cbcb2e41c817ffe3d5264c5b0f228" gracePeriod=30 Oct 30 13:39:53.586929 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:53.586891 4931 generic.go:356] "Generic (PLEG): container finished" podID="80f8171d-05ef-44b2-af37-12ea086b0334" containerID="797ebd0f0981f347f9c69624b9368825622cbcb2e41c817ffe3d5264c5b0f228" exitCode=0 Oct 30 13:39:53.593406 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:53.590383 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d" event={"ID":"80f8171d-05ef-44b2-af37-12ea086b0334","Type":"ContainerDied","Data":"797ebd0f0981f347f9c69624b9368825622cbcb2e41c817ffe3d5264c5b0f228"} Oct 30 13:39:53.610246 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:53.610147 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d" Oct 30 13:39:53.719315 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:53.713906 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80f8171d-05ef-44b2-af37-12ea086b0334-kube-api-access-qgxb2\" (UniqueName: \"kubernetes.io/projected/80f8171d-05ef-44b2-af37-12ea086b0334-kube-api-access-qgxb2\") pod \"80f8171d-05ef-44b2-af37-12ea086b0334\" (UID: \"80f8171d-05ef-44b2-af37-12ea086b0334\") " Oct 30 13:39:53.719315 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:53.713995 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80f8171d-05ef-44b2-af37-12ea086b0334-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f8171d-05ef-44b2-af37-12ea086b0334-openshift-service-ca\") pod \"80f8171d-05ef-44b2-af37-12ea086b0334\" (UID: \"80f8171d-05ef-44b2-af37-12ea086b0334\") " Oct 30 13:39:53.719315 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:53.714059 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80f8171d-05ef-44b2-af37-12ea086b0334-metrics-certs\" (UniqueName: \"kubernetes.io/secret/80f8171d-05ef-44b2-af37-12ea086b0334-metrics-certs\") pod \"80f8171d-05ef-44b2-af37-12ea086b0334\" (UID: \"80f8171d-05ef-44b2-af37-12ea086b0334\") " Oct 30 13:39:53.719315 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:53.714126 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80f8171d-05ef-44b2-af37-12ea086b0334-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80f8171d-05ef-44b2-af37-12ea086b0334-maas-api-tls\") pod \"80f8171d-05ef-44b2-af37-12ea086b0334\" (UID: \"80f8171d-05ef-44b2-af37-12ea086b0334\") " Oct 30 13:39:53.720079 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:53.720048 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80f8171d-05ef-44b2-af37-12ea086b0334-openshift-service-ca" pod "80f8171d-05ef-44b2-af37-12ea086b0334" (UID: "80f8171d-05ef-44b2-af37-12ea086b0334"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:39:53.724120 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:53.724091 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80f8171d-05ef-44b2-af37-12ea086b0334-metrics-certs" pod "80f8171d-05ef-44b2-af37-12ea086b0334" (UID: "80f8171d-05ef-44b2-af37-12ea086b0334"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:39:53.742340 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:53.742255 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f8171d-05ef-44b2-af37-12ea086b0334-kube-api-access-qgxb2" pod "80f8171d-05ef-44b2-af37-12ea086b0334" (UID: "80f8171d-05ef-44b2-af37-12ea086b0334"). InnerVolumeSpecName "kube-api-access-qgxb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:53.745222 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:53.745178 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80f8171d-05ef-44b2-af37-12ea086b0334-maas-api-tls" pod "80f8171d-05ef-44b2-af37-12ea086b0334" (UID: "80f8171d-05ef-44b2-af37-12ea086b0334"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:39:53.817045 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:53.816965 4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f8171d-05ef-44b2-af37-12ea086b0334-openshift-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:53.817045 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:53.816996 4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80f8171d-05ef-44b2-af37-12ea086b0334-metrics-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:53.817045 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:53.817006 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80f8171d-05ef-44b2-af37-12ea086b0334-maas-api-tls\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:53.817045 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:53.817014 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgxb2\" (UniqueName: \"kubernetes.io/projected/80f8171d-05ef-44b2-af37-12ea086b0334-kube-api-access-qgxb2\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:39:54.598471 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:54.598379 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d" Oct 30 13:39:54.598471 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:54.598418 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d" event={"ID":"80f8171d-05ef-44b2-af37-12ea086b0334","Type":"ContainerDied","Data":"cf40229e67d496020e19cf67d139fe60822dd4c0e04f7ebf2a3e8b3c0559a67a"} Oct 30 13:39:54.599008 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:54.598498 4931 scope.go:122] "RemoveContainer" containerID="797ebd0f0981f347f9c69624b9368825622cbcb2e41c817ffe3d5264c5b0f228" Oct 30 13:39:54.660051 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:54.660023 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d"] Oct 30 13:39:54.673482 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:54.673456 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a71ca-5576695f45-7rp7d"] Oct 30 13:39:55.513651 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:55.513616 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80f8171d-05ef-44b2-af37-12ea086b0334" path="/var/lib/kubelet/pods/80f8171d-05ef-44b2-af37-12ea086b0334/volumes" Oct 30 13:39:57.242002 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:57.241957 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-bcd76c-5qgjz"] Oct 30 13:39:57.252860 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:57.252816 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-bcd76c-5qgjz"] Oct 30 13:39:57.514320 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:57.514232 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea8ce722-998e-4d37-8e05-8e659833303e" path="/var/lib/kubelet/pods/ea8ce722-998e-4d37-8e05-8e659833303e/volumes" Oct 30 13:39:58.517313 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:58.517254 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3e8ee4-99stc"] Oct 30 13:39:58.517820 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:58.517784 4931 cpu_manager.go:414] "containerMap: removing container" podUID="80f8171d-05ef-44b2-af37-12ea086b0334" containerName="maas-api" Oct 30 13:39:58.517820 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:58.517797 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80f8171d-05ef-44b2-af37-12ea086b0334" containerName="maas-api" Oct 30 13:39:58.517820 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:58.517812 4931 cpu_manager.go:414] "containerMap: removing container" podUID="ea8ce722-998e-4d37-8e05-8e659833303e" containerName="revoke-keys" Oct 30 13:39:58.517820 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:58.517817 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea8ce722-998e-4d37-8e05-8e659833303e" containerName="revoke-keys" Oct 30 13:39:58.518352 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:58.517831 4931 cpu_manager.go:414] "containerMap: removing container" podUID="24214a0a-42ce-4fa2-b4ef-9a28771278ce" containerName="istio-proxy" Oct 30 13:39:58.518352 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:58.517837 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24214a0a-42ce-4fa2-b4ef-9a28771278ce" containerName="istio-proxy" Oct 30 13:39:58.518352 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:58.517859 4931 cpu_manager.go:414] "containerMap: removing container" podUID="e46deba0-1ef3-4b54-9821-d4fbf394c9a6" containerName="istio-proxy" Oct 30 13:39:58.518352 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:58.517864 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e46deba0-1ef3-4b54-9821-d4fbf394c9a6" containerName="istio-proxy" Oct 30 13:39:58.518352 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:58.517945 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="e46deba0-1ef3-4b54-9821-d4fbf394c9a6" containerName="istio-proxy" Oct 30 13:39:58.518352 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:58.517958 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="80f8171d-05ef-44b2-af37-12ea086b0334" containerName="maas-api" Oct 30 13:39:58.518352 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:58.517966 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea8ce722-998e-4d37-8e05-8e659833303e" containerName="revoke-keys" Oct 30 13:39:58.518352 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:58.517973 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="24214a0a-42ce-4fa2-b4ef-9a28771278ce" containerName="istio-proxy" Oct 30 13:39:58.522594 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:58.522564 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3e8ee4-99stc" Oct 30 13:39:58.545312 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:58.545282 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3e8ee4-99stc"] Oct 30 13:39:58.553808 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:58.553782 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:39:58.553973 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:58.553792 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:39:58.592935 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:58.592897 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1acaa648-8bf9-43de-8853-c612023405c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3e8ee4-99stc\" (UID: \"1acaa648-8bf9-43de-8853-c612023405c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3e8ee4-99stc" Oct 30 13:39:58.593099 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:58.593075 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1acaa648-8bf9-43de-8853-c612023405c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3e8ee4-99stc\" (UID: \"1acaa648-8bf9-43de-8853-c612023405c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3e8ee4-99stc" Oct 30 13:39:58.593155 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:58.593121 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jqts\" (UniqueName: \"kubernetes.io/projected/1acaa648-8bf9-43de-8853-c612023405c1-kube-api-access-4jqts\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3e8ee4-99stc\" (UID: \"1acaa648-8bf9-43de-8853-c612023405c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3e8ee4-99stc" Oct 30 13:39:58.694381 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:58.694341 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1acaa648-8bf9-43de-8853-c612023405c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3e8ee4-99stc\" (UID: \"1acaa648-8bf9-43de-8853-c612023405c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3e8ee4-99stc" Oct 30 13:39:58.694554 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:58.694476 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1acaa648-8bf9-43de-8853-c612023405c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3e8ee4-99stc\" (UID: \"1acaa648-8bf9-43de-8853-c612023405c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3e8ee4-99stc" Oct 30 13:39:58.694554 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:58.694518 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jqts\" (UniqueName: \"kubernetes.io/projected/1acaa648-8bf9-43de-8853-c612023405c1-kube-api-access-4jqts\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3e8ee4-99stc\" (UID: \"1acaa648-8bf9-43de-8853-c612023405c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3e8ee4-99stc" Oct 30 13:39:58.695103 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:58.695075 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1acaa648-8bf9-43de-8853-c612023405c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3e8ee4-99stc\" (UID: \"1acaa648-8bf9-43de-8853-c612023405c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3e8ee4-99stc" Oct 30 13:39:58.696844 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:58.696819 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1acaa648-8bf9-43de-8853-c612023405c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3e8ee4-99stc\" (UID: \"1acaa648-8bf9-43de-8853-c612023405c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3e8ee4-99stc" Oct 30 13:39:58.720798 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:58.720761 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jqts\" (UniqueName: \"kubernetes.io/projected/1acaa648-8bf9-43de-8853-c612023405c1-kube-api-access-4jqts\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3e8ee4-99stc\" (UID: \"1acaa648-8bf9-43de-8853-c612023405c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3e8ee4-99stc" Oct 30 13:39:58.837062 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:58.836983 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3e8ee4-99stc" Oct 30 13:39:59.149385 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:59.147934 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg"] Oct 30 13:39:59.149385 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:59.148201 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" podUID="6ba44c24-d5d3-461a-b2b3-bca6e225185c" containerName="istio-proxy" containerID="cri-o://d05dd6d57a2a681c908fc5e3e9b045013d307d852b1ba7c250b09ebe78f615cf" gracePeriod=30 Oct 30 13:39:59.177194 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:39:59.177165 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1acaa648_8bf9_43de_8853_c612023405c1.slice/crio-e96612c8036f5c57ae91e47cb7297b793446230a7f5131ef77e889f8b53967b0 WatchSource:0}: Error finding container e96612c8036f5c57ae91e47cb7297b793446230a7f5131ef77e889f8b53967b0: Status 404 returned error can't find the container with id e96612c8036f5c57ae91e47cb7297b793446230a7f5131ef77e889f8b53967b0 Oct 30 13:39:59.183056 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:59.183029 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3e8ee4-99stc"] Oct 30 13:39:59.641968 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:59.641934 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3e8ee4-99stc" event={"ID":"1acaa648-8bf9-43de-8853-c612023405c1","Type":"ContainerStarted","Data":"b8cc47f72b5a8fa9e7e8321ab9ea3ff124774cd05722e070037a1f9af57c3585"} Oct 30 13:39:59.642431 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:59.641988 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3e8ee4-99stc" event={"ID":"1acaa648-8bf9-43de-8853-c612023405c1","Type":"ContainerStarted","Data":"e96612c8036f5c57ae91e47cb7297b793446230a7f5131ef77e889f8b53967b0"} Oct 30 13:39:59.690062 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:59.690002 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3e8ee4-99stc" podStartSLOduration=1.689987849 podStartE2EDuration="1.689987849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:39:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:39:59.688636251 +0000 UTC m=+3298.292134710" watchObservedRunningTime="2026-10-30 13:39:59.689987849 +0000 UTC m=+3298.293486289" Oct 30 13:39:59.850527 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:39:59.850440 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:40:00.345612 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:00.345570 4931 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:40:00.345796 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:00.345632 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" podUID="6ba44c24-d5d3-461a-b2b3-bca6e225185c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:40:00.651536 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:00.651499 4931 generic.go:356] "Generic (PLEG): container finished" podID="1acaa648-8bf9-43de-8853-c612023405c1" containerID="b8cc47f72b5a8fa9e7e8321ab9ea3ff124774cd05722e070037a1f9af57c3585" exitCode=0 Oct 30 13:40:00.651982 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:00.651580 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3e8ee4-99stc" event={"ID":"1acaa648-8bf9-43de-8853-c612023405c1","Type":"ContainerDied","Data":"b8cc47f72b5a8fa9e7e8321ab9ea3ff124774cd05722e070037a1f9af57c3585"} Oct 30 13:40:01.842072 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:01.842042 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3e8ee4-99stc" Oct 30 13:40:01.941782 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:01.941709 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1acaa648-8bf9-43de-8853-c612023405c1-kube-api-access-4jqts\" (UniqueName: \"kubernetes.io/projected/1acaa648-8bf9-43de-8853-c612023405c1-kube-api-access-4jqts\") pod \"1acaa648-8bf9-43de-8853-c612023405c1\" (UID: \"1acaa648-8bf9-43de-8853-c612023405c1\") " Oct 30 13:40:01.941958 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:01.941938 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1acaa648-8bf9-43de-8853-c612023405c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1acaa648-8bf9-43de-8853-c612023405c1-maas-api-xks-ca\") pod \"1acaa648-8bf9-43de-8853-c612023405c1\" (UID: \"1acaa648-8bf9-43de-8853-c612023405c1\") " Oct 30 13:40:01.942065 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:01.942045 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1acaa648-8bf9-43de-8853-c612023405c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1acaa648-8bf9-43de-8853-c612023405c1-maas-api-openshift-ca\") pod \"1acaa648-8bf9-43de-8853-c612023405c1\" (UID: \"1acaa648-8bf9-43de-8853-c612023405c1\") " Oct 30 13:40:01.942436 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:01.942407 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1acaa648-8bf9-43de-8853-c612023405c1-maas-api-openshift-ca" pod "1acaa648-8bf9-43de-8853-c612023405c1" (UID: "1acaa648-8bf9-43de-8853-c612023405c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:40:01.942621 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:01.942604 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1acaa648-8bf9-43de-8853-c612023405c1-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:01.943993 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:01.943964 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1acaa648-8bf9-43de-8853-c612023405c1-kube-api-access-4jqts" pod "1acaa648-8bf9-43de-8853-c612023405c1" (UID: "1acaa648-8bf9-43de-8853-c612023405c1"). InnerVolumeSpecName "kube-api-access-4jqts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:01.944520 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:01.944490 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1acaa648-8bf9-43de-8853-c612023405c1-maas-api-xks-ca" pod "1acaa648-8bf9-43de-8853-c612023405c1" (UID: "1acaa648-8bf9-43de-8853-c612023405c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:40:02.044173 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:02.044132 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1acaa648-8bf9-43de-8853-c612023405c1-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:02.044173 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:02.044169 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jqts\" (UniqueName: \"kubernetes.io/projected/1acaa648-8bf9-43de-8853-c612023405c1-kube-api-access-4jqts\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:02.664114 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:02.664081 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/2.log" Oct 30 13:40:02.664626 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:02.664600 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/2.log" Oct 30 13:40:02.668364 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:02.668342 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3e8ee4-99stc" Oct 30 13:40:02.668678 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:02.668354 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3e8ee4-99stc" event={"ID":"1acaa648-8bf9-43de-8853-c612023405c1","Type":"ContainerDied","Data":"e96612c8036f5c57ae91e47cb7297b793446230a7f5131ef77e889f8b53967b0"} Oct 30 13:40:02.668947 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:02.668923 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e96612c8036f5c57ae91e47cb7297b793446230a7f5131ef77e889f8b53967b0" Oct 30 13:40:04.431052 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:04.430993 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" Oct 30 13:40:04.462791 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:04.462757 4931 scope.go:122] "RemoveContainer" containerID="5818472411fbdb6fe85ea3df7c0044ccb9f4f25d61bf7fe97baec779d9a1e93a" Oct 30 13:40:04.473790 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:04.473768 4931 scope.go:122] "RemoveContainer" containerID="d05dd6d57a2a681c908fc5e3e9b045013d307d852b1ba7c250b09ebe78f615cf" Oct 30 13:40:04.577797 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:04.577773 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ba44c24-d5d3-461a-b2b3-bca6e225185c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ba44c24-d5d3-461a-b2b3-bca6e225185c-istio-podinfo\") pod \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " Oct 30 13:40:04.577903 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:04.577810 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba44c24-d5d3-461a-b2b3-bca6e225185c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ba44c24-d5d3-461a-b2b3-bca6e225185c-istio-envoy\") pod \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " Oct 30 13:40:04.577903 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:04.577836 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ba44c24-d5d3-461a-b2b3-bca6e225185c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ba44c24-d5d3-461a-b2b3-bca6e225185c-istiod-ca-cert\") pod \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " Oct 30 13:40:04.577903 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:04.577872 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ba44c24-d5d3-461a-b2b3-bca6e225185c-istio-token\" (UniqueName: \"kubernetes.io/projected/6ba44c24-d5d3-461a-b2b3-bca6e225185c-istio-token\") pod \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " Oct 30 13:40:04.577903 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:04.577893 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ba44c24-d5d3-461a-b2b3-bca6e225185c-kube-api-access-blnwg\" (UniqueName: \"kubernetes.io/projected/6ba44c24-d5d3-461a-b2b3-bca6e225185c-kube-api-access-blnwg\") pod \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " Oct 30 13:40:04.578104 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:04.578036 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba44c24-d5d3-461a-b2b3-bca6e225185c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba44c24-d5d3-461a-b2b3-bca6e225185c-workload-certs\") pod \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " Oct 30 13:40:04.578219 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:04.578194 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba44c24-d5d3-461a-b2b3-bca6e225185c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba44c24-d5d3-461a-b2b3-bca6e225185c-workload-socket\") pod \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " Oct 30 13:40:04.578373 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:04.578286 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ba44c24-d5d3-461a-b2b3-bca6e225185c-istiod-ca-cert" pod "6ba44c24-d5d3-461a-b2b3-bca6e225185c" (UID: "6ba44c24-d5d3-461a-b2b3-bca6e225185c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:40:04.578373 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:04.578316 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba44c24-d5d3-461a-b2b3-bca6e225185c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ba44c24-d5d3-461a-b2b3-bca6e225185c-istio-data\") pod \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " Oct 30 13:40:04.578513 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:04.578382 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba44c24-d5d3-461a-b2b3-bca6e225185c-workload-certs" pod "6ba44c24-d5d3-461a-b2b3-bca6e225185c" (UID: "6ba44c24-d5d3-461a-b2b3-bca6e225185c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:04.578513 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:04.578416 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba44c24-d5d3-461a-b2b3-bca6e225185c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba44c24-d5d3-461a-b2b3-bca6e225185c-credential-socket\") pod \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\" (UID: \"6ba44c24-d5d3-461a-b2b3-bca6e225185c\") " Oct 30 13:40:04.578631 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:04.578550 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba44c24-d5d3-461a-b2b3-bca6e225185c-workload-socket" pod "6ba44c24-d5d3-461a-b2b3-bca6e225185c" (UID: "6ba44c24-d5d3-461a-b2b3-bca6e225185c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:04.578694 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:04.578643 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba44c24-d5d3-461a-b2b3-bca6e225185c-istio-data" pod "6ba44c24-d5d3-461a-b2b3-bca6e225185c" (UID: "6ba44c24-d5d3-461a-b2b3-bca6e225185c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:04.578805 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:04.578784 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba44c24-d5d3-461a-b2b3-bca6e225185c-credential-socket" pod "6ba44c24-d5d3-461a-b2b3-bca6e225185c" (UID: "6ba44c24-d5d3-461a-b2b3-bca6e225185c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:04.579019 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:04.578994 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba44c24-d5d3-461a-b2b3-bca6e225185c-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:04.579117 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:04.579025 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba44c24-d5d3-461a-b2b3-bca6e225185c-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:04.579117 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:04.579040 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ba44c24-d5d3-461a-b2b3-bca6e225185c-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:04.579117 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:04.579056 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba44c24-d5d3-461a-b2b3-bca6e225185c-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:04.579117 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:04.579071 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ba44c24-d5d3-461a-b2b3-bca6e225185c-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:04.580305 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:04.580252 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ba44c24-d5d3-461a-b2b3-bca6e225185c-istio-podinfo" pod "6ba44c24-d5d3-461a-b2b3-bca6e225185c" (UID: "6ba44c24-d5d3-461a-b2b3-bca6e225185c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:40:04.580429 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:04.580333 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba44c24-d5d3-461a-b2b3-bca6e225185c-istio-token" pod "6ba44c24-d5d3-461a-b2b3-bca6e225185c" (UID: "6ba44c24-d5d3-461a-b2b3-bca6e225185c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:04.580495 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:04.580445 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba44c24-d5d3-461a-b2b3-bca6e225185c-istio-envoy" pod "6ba44c24-d5d3-461a-b2b3-bca6e225185c" (UID: "6ba44c24-d5d3-461a-b2b3-bca6e225185c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:04.580495 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:04.580465 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba44c24-d5d3-461a-b2b3-bca6e225185c-kube-api-access-blnwg" pod "6ba44c24-d5d3-461a-b2b3-bca6e225185c" (UID: "6ba44c24-d5d3-461a-b2b3-bca6e225185c"). InnerVolumeSpecName "kube-api-access-blnwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:04.680840 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:04.680801 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ba44c24-d5d3-461a-b2b3-bca6e225185c-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:04.680840 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:04.680839 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ba44c24-d5d3-461a-b2b3-bca6e225185c-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:04.681198 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:04.680855 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ba44c24-d5d3-461a-b2b3-bca6e225185c-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:04.681198 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:04.680869 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blnwg\" (UniqueName: \"kubernetes.io/projected/6ba44c24-d5d3-461a-b2b3-bca6e225185c-kube-api-access-blnwg\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:04.684723 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:04.684680 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" event={"ID":"6ba44c24-d5d3-461a-b2b3-bca6e225185c","Type":"ContainerDied","Data":"d05dd6d57a2a681c908fc5e3e9b045013d307d852b1ba7c250b09ebe78f615cf"} Oct 30 13:40:04.684841 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:04.684741 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" Oct 30 13:40:04.684841 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:04.684755 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg" event={"ID":"6ba44c24-d5d3-461a-b2b3-bca6e225185c","Type":"ContainerDied","Data":"3169f4914961b5da9d01ba12ccfd59be05c0f92376e8905ef8107e9f24cfe0fb"} Oct 30 13:40:04.789537 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:04.789499 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg"] Oct 30 13:40:04.792409 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:04.792386 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg"] Oct 30 13:40:05.519817 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:05.519778 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ba44c24-d5d3-461a-b2b3-bca6e225185c" path="/var/lib/kubelet/pods/6ba44c24-d5d3-461a-b2b3-bca6e225185c/volumes" Oct 30 13:40:06.001087 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:06.001048 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg"] Oct 30 13:40:06.001387 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:06.001353 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" podUID="8a110d96-ba95-4582-9e51-321f91ea6510" containerName="istio-proxy" containerID="cri-o://59bc6d872406f1fd6f13b11ce1850fd8f7e77be1336e8994d4b5ceb9a3dccdd7" gracePeriod=30 Oct 30 13:40:06.005950 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:06.005922 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx"] Oct 30 13:40:06.006202 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:06.006177 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" podUID="defbc903-546b-4dfd-a284-50e517429e6f" containerName="istio-proxy" containerID="cri-o://eb83281d54ca11df15a7cede8ac4475182b4a01550a5bbee6d7d379e49c02242" gracePeriod=30 Oct 30 13:40:06.775222 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:06.775168 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c90b9e-hr6ws"] Oct 30 13:40:06.775970 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:06.775944 4931 cpu_manager.go:414] "containerMap: removing container" podUID="6ba44c24-d5d3-461a-b2b3-bca6e225185c" containerName="istio-proxy" Oct 30 13:40:06.776063 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:06.775972 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ba44c24-d5d3-461a-b2b3-bca6e225185c" containerName="istio-proxy" Oct 30 13:40:06.776063 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:06.775989 4931 cpu_manager.go:414] "containerMap: removing container" podUID="1acaa648-8bf9-43de-8853-c612023405c1" containerName="revoke-keys" Oct 30 13:40:06.776063 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:06.775996 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1acaa648-8bf9-43de-8853-c612023405c1" containerName="revoke-keys" Oct 30 13:40:06.776221 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:06.776124 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ba44c24-d5d3-461a-b2b3-bca6e225185c" containerName="istio-proxy" Oct 30 13:40:06.776221 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:06.776140 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="1acaa648-8bf9-43de-8853-c612023405c1" containerName="revoke-keys" Oct 30 13:40:06.783877 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:06.783840 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c90b9e-hr6ws" Oct 30 13:40:06.800390 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:06.800353 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c90b9e-hr6ws"] Oct 30 13:40:06.813746 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:06.813694 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:40:06.813966 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:06.813881 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:40:06.848208 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:06.847072 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a57f38-0527-4855-b62d-2e18d16bf760-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c90b9e-hr6ws\" (UID: \"06a57f38-0527-4855-b62d-2e18d16bf760\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c90b9e-hr6ws" Oct 30 13:40:06.848208 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:06.847122 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fplc4\" (UniqueName: \"kubernetes.io/projected/06a57f38-0527-4855-b62d-2e18d16bf760-kube-api-access-fplc4\") pod \"maas-api-revoke-keys-e2e-worker-w0-c90b9e-hr6ws\" (UID: \"06a57f38-0527-4855-b62d-2e18d16bf760\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c90b9e-hr6ws" Oct 30 13:40:06.848208 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:06.847170 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a57f38-0527-4855-b62d-2e18d16bf760-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c90b9e-hr6ws\" (UID: \"06a57f38-0527-4855-b62d-2e18d16bf760\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c90b9e-hr6ws" Oct 30 13:40:06.948538 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:06.948437 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a57f38-0527-4855-b62d-2e18d16bf760-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c90b9e-hr6ws\" (UID: \"06a57f38-0527-4855-b62d-2e18d16bf760\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c90b9e-hr6ws" Oct 30 13:40:06.948538 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:06.948475 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fplc4\" (UniqueName: \"kubernetes.io/projected/06a57f38-0527-4855-b62d-2e18d16bf760-kube-api-access-fplc4\") pod \"maas-api-revoke-keys-e2e-worker-w0-c90b9e-hr6ws\" (UID: \"06a57f38-0527-4855-b62d-2e18d16bf760\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c90b9e-hr6ws" Oct 30 13:40:06.948538 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:06.948512 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a57f38-0527-4855-b62d-2e18d16bf760-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c90b9e-hr6ws\" (UID: \"06a57f38-0527-4855-b62d-2e18d16bf760\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c90b9e-hr6ws" Oct 30 13:40:06.949344 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:06.949320 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a57f38-0527-4855-b62d-2e18d16bf760-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c90b9e-hr6ws\" (UID: \"06a57f38-0527-4855-b62d-2e18d16bf760\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c90b9e-hr6ws" Oct 30 13:40:06.951768 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:06.951731 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a57f38-0527-4855-b62d-2e18d16bf760-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c90b9e-hr6ws\" (UID: \"06a57f38-0527-4855-b62d-2e18d16bf760\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c90b9e-hr6ws" Oct 30 13:40:06.978825 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:06.978786 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fplc4\" (UniqueName: \"kubernetes.io/projected/06a57f38-0527-4855-b62d-2e18d16bf760-kube-api-access-fplc4\") pod \"maas-api-revoke-keys-e2e-worker-w0-c90b9e-hr6ws\" (UID: \"06a57f38-0527-4855-b62d-2e18d16bf760\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c90b9e-hr6ws" Oct 30 13:40:07.101187 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:07.101084 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c90b9e-hr6ws" Oct 30 13:40:07.376515 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:40:07.376430 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:40:07.421549 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:07.421515 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c90b9e-hr6ws"] Oct 30 13:40:07.423776 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:40:07.423745 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06a57f38_0527_4855_b62d_2e18d16bf760.slice/crio-abcadcc4da6f23715760d8f833f388806aa01bad69078f2df76621702f074f92 WatchSource:0}: Error finding container abcadcc4da6f23715760d8f833f388806aa01bad69078f2df76621702f074f92: Status 404 returned error can't find the container with id abcadcc4da6f23715760d8f833f388806aa01bad69078f2df76621702f074f92 Oct 30 13:40:07.735357 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:07.735317 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c90b9e-hr6ws" event={"ID":"06a57f38-0527-4855-b62d-2e18d16bf760","Type":"ContainerStarted","Data":"abcadcc4da6f23715760d8f833f388806aa01bad69078f2df76621702f074f92"} Oct 30 13:40:08.746515 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:08.746480 4931 generic.go:356] "Generic (PLEG): container finished" podID="06a57f38-0527-4855-b62d-2e18d16bf760" containerID="a25b0f2a4a9e72c2e954925993cbedecb3b64b5499a34bb159c50d0325190aa0" exitCode=0 Oct 30 13:40:08.747019 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:08.746555 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c90b9e-hr6ws" event={"ID":"06a57f38-0527-4855-b62d-2e18d16bf760","Type":"ContainerDied","Data":"a25b0f2a4a9e72c2e954925993cbedecb3b64b5499a34bb159c50d0325190aa0"} Oct 30 13:40:08.789789 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:08.789755 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk"] Oct 30 13:40:08.798530 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:08.798504 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84"] Oct 30 13:40:09.968673 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:09.968641 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c90b9e-hr6ws" Oct 30 13:40:09.991562 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:09.991532 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06a57f38-0527-4855-b62d-2e18d16bf760-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a57f38-0527-4855-b62d-2e18d16bf760-maas-api-xks-ca\") pod \"06a57f38-0527-4855-b62d-2e18d16bf760\" (UID: \"06a57f38-0527-4855-b62d-2e18d16bf760\") " Oct 30 13:40:09.991713 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:09.991618 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06a57f38-0527-4855-b62d-2e18d16bf760-kube-api-access-fplc4\" (UniqueName: \"kubernetes.io/projected/06a57f38-0527-4855-b62d-2e18d16bf760-kube-api-access-fplc4\") pod \"06a57f38-0527-4855-b62d-2e18d16bf760\" (UID: \"06a57f38-0527-4855-b62d-2e18d16bf760\") " Oct 30 13:40:09.991713 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:09.991652 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06a57f38-0527-4855-b62d-2e18d16bf760-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a57f38-0527-4855-b62d-2e18d16bf760-maas-api-openshift-ca\") pod \"06a57f38-0527-4855-b62d-2e18d16bf760\" (UID: \"06a57f38-0527-4855-b62d-2e18d16bf760\") " Oct 30 13:40:09.992073 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:09.992040 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06a57f38-0527-4855-b62d-2e18d16bf760-maas-api-openshift-ca" pod "06a57f38-0527-4855-b62d-2e18d16bf760" (UID: "06a57f38-0527-4855-b62d-2e18d16bf760"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:40:09.993669 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:09.993648 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06a57f38-0527-4855-b62d-2e18d16bf760-maas-api-xks-ca" pod "06a57f38-0527-4855-b62d-2e18d16bf760" (UID: "06a57f38-0527-4855-b62d-2e18d16bf760"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:40:09.993849 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:09.993828 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06a57f38-0527-4855-b62d-2e18d16bf760-kube-api-access-fplc4" pod "06a57f38-0527-4855-b62d-2e18d16bf760" (UID: "06a57f38-0527-4855-b62d-2e18d16bf760"). InnerVolumeSpecName "kube-api-access-fplc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:10.093036 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:10.092941 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a57f38-0527-4855-b62d-2e18d16bf760-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:10.093036 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:10.092976 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fplc4\" (UniqueName: \"kubernetes.io/projected/06a57f38-0527-4855-b62d-2e18d16bf760-kube-api-access-fplc4\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:10.093036 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:10.092990 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a57f38-0527-4855-b62d-2e18d16bf760-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:10.762741 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:10.762663 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c90b9e-hr6ws" event={"ID":"06a57f38-0527-4855-b62d-2e18d16bf760","Type":"ContainerDied","Data":"abcadcc4da6f23715760d8f833f388806aa01bad69078f2df76621702f074f92"} Oct 30 13:40:10.762741 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:10.762688 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c90b9e-hr6ws" Oct 30 13:40:10.762741 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:10.762716 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abcadcc4da6f23715760d8f833f388806aa01bad69078f2df76621702f074f92" Oct 30 13:40:11.287528 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.287502 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" Oct 30 13:40:11.308351 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.308322 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a110d96-ba95-4582-9e51-321f91ea6510-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a110d96-ba95-4582-9e51-321f91ea6510-istio-envoy\") pod \"8a110d96-ba95-4582-9e51-321f91ea6510\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " Oct 30 13:40:11.308487 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.308383 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a110d96-ba95-4582-9e51-321f91ea6510-kube-api-access-vnzdl\" (UniqueName: \"kubernetes.io/projected/8a110d96-ba95-4582-9e51-321f91ea6510-kube-api-access-vnzdl\") pod \"8a110d96-ba95-4582-9e51-321f91ea6510\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " Oct 30 13:40:11.308487 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.308434 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a110d96-ba95-4582-9e51-321f91ea6510-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a110d96-ba95-4582-9e51-321f91ea6510-credential-socket\") pod \"8a110d96-ba95-4582-9e51-321f91ea6510\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " Oct 30 13:40:11.308629 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.308488 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a110d96-ba95-4582-9e51-321f91ea6510-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a110d96-ba95-4582-9e51-321f91ea6510-istiod-ca-cert\") pod \"8a110d96-ba95-4582-9e51-321f91ea6510\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " Oct 30 13:40:11.308629 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.308571 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a110d96-ba95-4582-9e51-321f91ea6510-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a110d96-ba95-4582-9e51-321f91ea6510-istio-data\") pod \"8a110d96-ba95-4582-9e51-321f91ea6510\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " Oct 30 13:40:11.308739 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.308667 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a110d96-ba95-4582-9e51-321f91ea6510-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a110d96-ba95-4582-9e51-321f91ea6510-workload-certs\") pod \"8a110d96-ba95-4582-9e51-321f91ea6510\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " Oct 30 13:40:11.308739 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.308734 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a110d96-ba95-4582-9e51-321f91ea6510-istio-token\" (UniqueName: \"kubernetes.io/projected/8a110d96-ba95-4582-9e51-321f91ea6510-istio-token\") pod \"8a110d96-ba95-4582-9e51-321f91ea6510\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " Oct 30 13:40:11.308866 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.308761 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a110d96-ba95-4582-9e51-321f91ea6510-credential-socket" pod "8a110d96-ba95-4582-9e51-321f91ea6510" (UID: "8a110d96-ba95-4582-9e51-321f91ea6510"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:11.308866 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.308783 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a110d96-ba95-4582-9e51-321f91ea6510-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a110d96-ba95-4582-9e51-321f91ea6510-workload-socket\") pod \"8a110d96-ba95-4582-9e51-321f91ea6510\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " Oct 30 13:40:11.308866 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.308816 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a110d96-ba95-4582-9e51-321f91ea6510-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a110d96-ba95-4582-9e51-321f91ea6510-istio-podinfo\") pod \"8a110d96-ba95-4582-9e51-321f91ea6510\" (UID: \"8a110d96-ba95-4582-9e51-321f91ea6510\") " Oct 30 13:40:11.308866 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.308831 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a110d96-ba95-4582-9e51-321f91ea6510-istio-data" pod "8a110d96-ba95-4582-9e51-321f91ea6510" (UID: "8a110d96-ba95-4582-9e51-321f91ea6510"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:11.309071 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.308903 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a110d96-ba95-4582-9e51-321f91ea6510-istiod-ca-cert" pod "8a110d96-ba95-4582-9e51-321f91ea6510" (UID: "8a110d96-ba95-4582-9e51-321f91ea6510"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:40:11.309174 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.309150 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a110d96-ba95-4582-9e51-321f91ea6510-workload-certs" pod "8a110d96-ba95-4582-9e51-321f91ea6510" (UID: "8a110d96-ba95-4582-9e51-321f91ea6510"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:11.309485 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.309461 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a110d96-ba95-4582-9e51-321f91ea6510-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:11.309574 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.309490 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a110d96-ba95-4582-9e51-321f91ea6510-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:11.309574 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.309503 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a110d96-ba95-4582-9e51-321f91ea6510-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:11.309574 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.309516 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a110d96-ba95-4582-9e51-321f91ea6510-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:11.309746 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.309594 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a110d96-ba95-4582-9e51-321f91ea6510-workload-socket" pod "8a110d96-ba95-4582-9e51-321f91ea6510" (UID: "8a110d96-ba95-4582-9e51-321f91ea6510"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:11.310706 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.310677 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a110d96-ba95-4582-9e51-321f91ea6510-istio-envoy" pod "8a110d96-ba95-4582-9e51-321f91ea6510" (UID: "8a110d96-ba95-4582-9e51-321f91ea6510"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:11.310931 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.310912 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a110d96-ba95-4582-9e51-321f91ea6510-kube-api-access-vnzdl" pod "8a110d96-ba95-4582-9e51-321f91ea6510" (UID: "8a110d96-ba95-4582-9e51-321f91ea6510"). InnerVolumeSpecName "kube-api-access-vnzdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:11.311074 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.311053 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a110d96-ba95-4582-9e51-321f91ea6510-istio-token" pod "8a110d96-ba95-4582-9e51-321f91ea6510" (UID: "8a110d96-ba95-4582-9e51-321f91ea6510"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:11.311310 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.311256 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a110d96-ba95-4582-9e51-321f91ea6510-istio-podinfo" pod "8a110d96-ba95-4582-9e51-321f91ea6510" (UID: "8a110d96-ba95-4582-9e51-321f91ea6510"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:40:11.313211 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.313192 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" Oct 30 13:40:11.410277 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.410229 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/defbc903-546b-4dfd-a284-50e517429e6f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/defbc903-546b-4dfd-a284-50e517429e6f-istio-podinfo\") pod \"defbc903-546b-4dfd-a284-50e517429e6f\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " Oct 30 13:40:11.410439 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.410388 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/defbc903-546b-4dfd-a284-50e517429e6f-kube-api-access-vbdtc\" (UniqueName: \"kubernetes.io/projected/defbc903-546b-4dfd-a284-50e517429e6f-kube-api-access-vbdtc\") pod \"defbc903-546b-4dfd-a284-50e517429e6f\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " Oct 30 13:40:11.410439 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.410432 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/defbc903-546b-4dfd-a284-50e517429e6f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/defbc903-546b-4dfd-a284-50e517429e6f-istiod-ca-cert\") pod \"defbc903-546b-4dfd-a284-50e517429e6f\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " Oct 30 13:40:11.410520 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.410463 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/defbc903-546b-4dfd-a284-50e517429e6f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/defbc903-546b-4dfd-a284-50e517429e6f-credential-socket\") pod \"defbc903-546b-4dfd-a284-50e517429e6f\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " Oct 30 13:40:11.410520 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.410499 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/defbc903-546b-4dfd-a284-50e517429e6f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/defbc903-546b-4dfd-a284-50e517429e6f-istio-envoy\") pod \"defbc903-546b-4dfd-a284-50e517429e6f\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " Oct 30 13:40:11.410620 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.410541 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/defbc903-546b-4dfd-a284-50e517429e6f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/defbc903-546b-4dfd-a284-50e517429e6f-istio-data\") pod \"defbc903-546b-4dfd-a284-50e517429e6f\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " Oct 30 13:40:11.410620 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.410567 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/defbc903-546b-4dfd-a284-50e517429e6f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/defbc903-546b-4dfd-a284-50e517429e6f-workload-socket\") pod \"defbc903-546b-4dfd-a284-50e517429e6f\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " Oct 30 13:40:11.410729 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.410619 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/defbc903-546b-4dfd-a284-50e517429e6f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/defbc903-546b-4dfd-a284-50e517429e6f-workload-certs\") pod \"defbc903-546b-4dfd-a284-50e517429e6f\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " Oct 30 13:40:11.410729 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.410715 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/defbc903-546b-4dfd-a284-50e517429e6f-istio-token\" (UniqueName: \"kubernetes.io/projected/defbc903-546b-4dfd-a284-50e517429e6f-istio-token\") pod \"defbc903-546b-4dfd-a284-50e517429e6f\" (UID: \"defbc903-546b-4dfd-a284-50e517429e6f\") " Oct 30 13:40:11.410856 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.410827 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/defbc903-546b-4dfd-a284-50e517429e6f-credential-socket" pod "defbc903-546b-4dfd-a284-50e517429e6f" (UID: "defbc903-546b-4dfd-a284-50e517429e6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:11.410977 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.410948 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/defbc903-546b-4dfd-a284-50e517429e6f-workload-socket" pod "defbc903-546b-4dfd-a284-50e517429e6f" (UID: "defbc903-546b-4dfd-a284-50e517429e6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:11.411041 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.410970 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/defbc903-546b-4dfd-a284-50e517429e6f-istiod-ca-cert" pod "defbc903-546b-4dfd-a284-50e517429e6f" (UID: "defbc903-546b-4dfd-a284-50e517429e6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:40:11.411109 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.411071 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/defbc903-546b-4dfd-a284-50e517429e6f-workload-certs" pod "defbc903-546b-4dfd-a284-50e517429e6f" (UID: "defbc903-546b-4dfd-a284-50e517429e6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:11.411381 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.411355 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a110d96-ba95-4582-9e51-321f91ea6510-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:11.411381 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.411382 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a110d96-ba95-4582-9e51-321f91ea6510-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:11.411633 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.411397 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a110d96-ba95-4582-9e51-321f91ea6510-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:11.411633 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.411411 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/defbc903-546b-4dfd-a284-50e517429e6f-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:11.411633 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.411424 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/defbc903-546b-4dfd-a284-50e517429e6f-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:11.411633 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.411437 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a110d96-ba95-4582-9e51-321f91ea6510-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:11.411633 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.411450 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/defbc903-546b-4dfd-a284-50e517429e6f-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:11.411633 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.411502 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnzdl\" (UniqueName: \"kubernetes.io/projected/8a110d96-ba95-4582-9e51-321f91ea6510-kube-api-access-vnzdl\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:11.411633 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.411519 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/defbc903-546b-4dfd-a284-50e517429e6f-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:11.411633 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.411589 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/defbc903-546b-4dfd-a284-50e517429e6f-istio-data" pod "defbc903-546b-4dfd-a284-50e517429e6f" (UID: "defbc903-546b-4dfd-a284-50e517429e6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:11.412773 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.412747 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/defbc903-546b-4dfd-a284-50e517429e6f-istio-podinfo" pod "defbc903-546b-4dfd-a284-50e517429e6f" (UID: "defbc903-546b-4dfd-a284-50e517429e6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:40:11.412977 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.412959 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/defbc903-546b-4dfd-a284-50e517429e6f-istio-token" pod "defbc903-546b-4dfd-a284-50e517429e6f" (UID: "defbc903-546b-4dfd-a284-50e517429e6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:11.413147 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.413125 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/defbc903-546b-4dfd-a284-50e517429e6f-kube-api-access-vbdtc" pod "defbc903-546b-4dfd-a284-50e517429e6f" (UID: "defbc903-546b-4dfd-a284-50e517429e6f"). InnerVolumeSpecName "kube-api-access-vbdtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:11.413344 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.413322 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/defbc903-546b-4dfd-a284-50e517429e6f-istio-envoy" pod "defbc903-546b-4dfd-a284-50e517429e6f" (UID: "defbc903-546b-4dfd-a284-50e517429e6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:11.498549 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.498509 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf"] Oct 30 13:40:11.512257 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.512226 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/defbc903-546b-4dfd-a284-50e517429e6f-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:11.512257 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.512254 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/defbc903-546b-4dfd-a284-50e517429e6f-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:11.512554 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.512278 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbdtc\" (UniqueName: \"kubernetes.io/projected/defbc903-546b-4dfd-a284-50e517429e6f-kube-api-access-vbdtc\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:11.512554 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.512289 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/defbc903-546b-4dfd-a284-50e517429e6f-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:11.512554 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.512297 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/defbc903-546b-4dfd-a284-50e517429e6f-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:11.550182 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.550153 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c"] Oct 30 13:40:11.770790 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.770758 4931 generic.go:356] "Generic (PLEG): container finished" podID="defbc903-546b-4dfd-a284-50e517429e6f" containerID="eb83281d54ca11df15a7cede8ac4475182b4a01550a5bbee6d7d379e49c02242" exitCode=0 Oct 30 13:40:11.770980 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.770820 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" Oct 30 13:40:11.770980 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.770849 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" event={"ID":"defbc903-546b-4dfd-a284-50e517429e6f","Type":"ContainerDied","Data":"eb83281d54ca11df15a7cede8ac4475182b4a01550a5bbee6d7d379e49c02242"} Oct 30 13:40:11.770980 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.770933 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx" event={"ID":"defbc903-546b-4dfd-a284-50e517429e6f","Type":"ContainerDied","Data":"64f03daa42576202fc59842299485a3b3f775a9be9efc267fef1a07565433440"} Oct 30 13:40:11.770980 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.770968 4931 scope.go:122] "RemoveContainer" containerID="eb83281d54ca11df15a7cede8ac4475182b4a01550a5bbee6d7d379e49c02242" Oct 30 13:40:11.773992 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.773953 4931 generic.go:356] "Generic (PLEG): container finished" podID="8a110d96-ba95-4582-9e51-321f91ea6510" containerID="59bc6d872406f1fd6f13b11ce1850fd8f7e77be1336e8994d4b5ceb9a3dccdd7" exitCode=0 Oct 30 13:40:11.774140 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.774081 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" event={"ID":"8a110d96-ba95-4582-9e51-321f91ea6510","Type":"ContainerDied","Data":"59bc6d872406f1fd6f13b11ce1850fd8f7e77be1336e8994d4b5ceb9a3dccdd7"} Oct 30 13:40:11.774140 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.774123 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" event={"ID":"8a110d96-ba95-4582-9e51-321f91ea6510","Type":"ContainerDied","Data":"17f493d53e5b456fb48d64bfb72ac8808078a26cd3a2fd88a3223fd2f832415d"} Oct 30 13:40:11.774262 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.774209 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg" Oct 30 13:40:11.791732 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.791615 4931 scope.go:122] "RemoveContainer" containerID="eb83281d54ca11df15a7cede8ac4475182b4a01550a5bbee6d7d379e49c02242" Oct 30 13:40:11.792504 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:40:11.792472 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb83281d54ca11df15a7cede8ac4475182b4a01550a5bbee6d7d379e49c02242\": container with ID starting with eb83281d54ca11df15a7cede8ac4475182b4a01550a5bbee6d7d379e49c02242 not found: ID does not exist" containerID="eb83281d54ca11df15a7cede8ac4475182b4a01550a5bbee6d7d379e49c02242" Oct 30 13:40:11.792605 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.792515 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb83281d54ca11df15a7cede8ac4475182b4a01550a5bbee6d7d379e49c02242"} err="failed to get container status \"eb83281d54ca11df15a7cede8ac4475182b4a01550a5bbee6d7d379e49c02242\": rpc error: code = NotFound desc = could not find container \"eb83281d54ca11df15a7cede8ac4475182b4a01550a5bbee6d7d379e49c02242\": container with ID starting with eb83281d54ca11df15a7cede8ac4475182b4a01550a5bbee6d7d379e49c02242 not found: ID does not exist" Oct 30 13:40:11.792605 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.792540 4931 scope.go:122] "RemoveContainer" containerID="59bc6d872406f1fd6f13b11ce1850fd8f7e77be1336e8994d4b5ceb9a3dccdd7" Oct 30 13:40:11.805759 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.805738 4931 scope.go:122] "RemoveContainer" containerID="59bc6d872406f1fd6f13b11ce1850fd8f7e77be1336e8994d4b5ceb9a3dccdd7" Oct 30 13:40:11.806623 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:40:11.806600 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59bc6d872406f1fd6f13b11ce1850fd8f7e77be1336e8994d4b5ceb9a3dccdd7\": container with ID starting with 59bc6d872406f1fd6f13b11ce1850fd8f7e77be1336e8994d4b5ceb9a3dccdd7 not found: ID does not exist" containerID="59bc6d872406f1fd6f13b11ce1850fd8f7e77be1336e8994d4b5ceb9a3dccdd7" Oct 30 13:40:11.806732 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.806628 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59bc6d872406f1fd6f13b11ce1850fd8f7e77be1336e8994d4b5ceb9a3dccdd7"} err="failed to get container status \"59bc6d872406f1fd6f13b11ce1850fd8f7e77be1336e8994d4b5ceb9a3dccdd7\": rpc error: code = NotFound desc = could not find container \"59bc6d872406f1fd6f13b11ce1850fd8f7e77be1336e8994d4b5ceb9a3dccdd7\": container with ID starting with 59bc6d872406f1fd6f13b11ce1850fd8f7e77be1336e8994d4b5ceb9a3dccdd7 not found: ID does not exist" Oct 30 13:40:11.862011 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.861972 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx"] Oct 30 13:40:11.877140 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.877113 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx"] Oct 30 13:40:11.934590 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.934558 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg"] Oct 30 13:40:11.941451 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:11.941422 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg"] Oct 30 13:40:13.513188 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:13.513149 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a110d96-ba95-4582-9e51-321f91ea6510" path="/var/lib/kubelet/pods/8a110d96-ba95-4582-9e51-321f91ea6510/volumes" Oct 30 13:40:13.513723 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:13.513704 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="defbc903-546b-4dfd-a284-50e517429e6f" path="/var/lib/kubelet/pods/defbc903-546b-4dfd-a284-50e517429e6f/volumes" Oct 30 13:40:19.470301 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:19.460402 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de97761d-tf5v5"] Oct 30 13:40:19.475289 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:19.471680 4931 cpu_manager.go:414] "containerMap: removing container" podUID="defbc903-546b-4dfd-a284-50e517429e6f" containerName="istio-proxy" Oct 30 13:40:19.509706 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:19.508873 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="defbc903-546b-4dfd-a284-50e517429e6f" containerName="istio-proxy" Oct 30 13:40:19.509706 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:19.508987 4931 cpu_manager.go:414] "containerMap: removing container" podUID="8a110d96-ba95-4582-9e51-321f91ea6510" containerName="istio-proxy" Oct 30 13:40:19.509706 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:19.508996 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a110d96-ba95-4582-9e51-321f91ea6510" containerName="istio-proxy" Oct 30 13:40:19.509706 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:19.509008 4931 cpu_manager.go:414] "containerMap: removing container" podUID="06a57f38-0527-4855-b62d-2e18d16bf760" containerName="revoke-keys" Oct 30 13:40:19.509706 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:19.509015 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06a57f38-0527-4855-b62d-2e18d16bf760" containerName="revoke-keys" Oct 30 13:40:19.509706 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:19.509219 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a110d96-ba95-4582-9e51-321f91ea6510" containerName="istio-proxy" Oct 30 13:40:19.509706 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:19.509233 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="06a57f38-0527-4855-b62d-2e18d16bf760" containerName="revoke-keys" Oct 30 13:40:19.509706 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:19.509247 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="defbc903-546b-4dfd-a284-50e517429e6f" containerName="istio-proxy" Oct 30 13:40:19.515984 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:19.515927 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de97761d-tf5v5" Oct 30 13:40:19.529134 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:19.529103 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de97761d-tf5v5"] Oct 30 13:40:19.566098 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:19.566062 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:40:19.566283 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:19.566144 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:40:19.619016 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:19.618981 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4m6f\" (UniqueName: \"kubernetes.io/projected/145f3cc9-6c5a-4c88-bc89-da9c956c224d-kube-api-access-d4m6f\") pod \"maas-api-revoke-sub-de97761d-tf5v5\" (UID: \"145f3cc9-6c5a-4c88-bc89-da9c956c224d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de97761d-tf5v5" Oct 30 13:40:19.619216 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:19.619028 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/145f3cc9-6c5a-4c88-bc89-da9c956c224d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de97761d-tf5v5\" (UID: \"145f3cc9-6c5a-4c88-bc89-da9c956c224d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de97761d-tf5v5" Oct 30 13:40:19.619216 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:19.619182 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/145f3cc9-6c5a-4c88-bc89-da9c956c224d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de97761d-tf5v5\" (UID: \"145f3cc9-6c5a-4c88-bc89-da9c956c224d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de97761d-tf5v5" Oct 30 13:40:19.720037 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:19.719988 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4m6f\" (UniqueName: \"kubernetes.io/projected/145f3cc9-6c5a-4c88-bc89-da9c956c224d-kube-api-access-d4m6f\") pod \"maas-api-revoke-sub-de97761d-tf5v5\" (UID: \"145f3cc9-6c5a-4c88-bc89-da9c956c224d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de97761d-tf5v5" Oct 30 13:40:19.720282 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:19.720047 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/145f3cc9-6c5a-4c88-bc89-da9c956c224d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de97761d-tf5v5\" (UID: \"145f3cc9-6c5a-4c88-bc89-da9c956c224d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de97761d-tf5v5" Oct 30 13:40:19.720282 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:19.720109 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/145f3cc9-6c5a-4c88-bc89-da9c956c224d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de97761d-tf5v5\" (UID: \"145f3cc9-6c5a-4c88-bc89-da9c956c224d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de97761d-tf5v5" Oct 30 13:40:19.720786 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:19.720728 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/145f3cc9-6c5a-4c88-bc89-da9c956c224d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de97761d-tf5v5\" (UID: \"145f3cc9-6c5a-4c88-bc89-da9c956c224d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de97761d-tf5v5" Oct 30 13:40:19.722920 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:19.722888 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/145f3cc9-6c5a-4c88-bc89-da9c956c224d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de97761d-tf5v5\" (UID: \"145f3cc9-6c5a-4c88-bc89-da9c956c224d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de97761d-tf5v5" Oct 30 13:40:19.745257 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:19.745230 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4m6f\" (UniqueName: \"kubernetes.io/projected/145f3cc9-6c5a-4c88-bc89-da9c956c224d-kube-api-access-d4m6f\") pod \"maas-api-revoke-sub-de97761d-tf5v5\" (UID: \"145f3cc9-6c5a-4c88-bc89-da9c956c224d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de97761d-tf5v5" Oct 30 13:40:19.847584 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:19.847550 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de97761d-tf5v5" Oct 30 13:40:19.999490 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:19.999402 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3e8ee4-99stc"] Oct 30 13:40:20.005647 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:20.005608 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3e8ee4-99stc"] Oct 30 13:40:20.162296 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:20.162246 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de97761d-tf5v5"] Oct 30 13:40:20.164036 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:40:20.164003 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod145f3cc9_6c5a_4c88_bc89_da9c956c224d.slice/crio-28fc6d7651e54637407bbad3562e4e0a67be2aad60933552767a48a3ec8892a0 WatchSource:0}: Error finding container 28fc6d7651e54637407bbad3562e4e0a67be2aad60933552767a48a3ec8892a0: Status 404 returned error can't find the container with id 28fc6d7651e54637407bbad3562e4e0a67be2aad60933552767a48a3ec8892a0 Oct 30 13:40:20.845128 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:20.845046 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de97761d-tf5v5" event={"ID":"145f3cc9-6c5a-4c88-bc89-da9c956c224d","Type":"ContainerStarted","Data":"f01a91434e52069c23159dc6c2cda66f5810b0cf54b393ffddc4270e0cc0d0ad"} Oct 30 13:40:20.845128 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:20.845100 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de97761d-tf5v5" event={"ID":"145f3cc9-6c5a-4c88-bc89-da9c956c224d","Type":"ContainerStarted","Data":"28fc6d7651e54637407bbad3562e4e0a67be2aad60933552767a48a3ec8892a0"} Oct 30 13:40:20.891117 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:20.891060 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de97761d-tf5v5" podStartSLOduration=1.8910466559999999 podStartE2EDuration="1.891046656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:20.889027263 +0000 UTC m=+3319.492525710" watchObservedRunningTime="2026-10-30 13:40:20.891046656 +0000 UTC m=+3319.494545101" Oct 30 13:40:21.513750 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:21.513678 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1acaa648-8bf9-43de-8853-c612023405c1" path="/var/lib/kubelet/pods/1acaa648-8bf9-43de-8853-c612023405c1/volumes" Oct 30 13:40:21.814055 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:21.813970 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5aa264b-ntw2b"] Oct 30 13:40:21.817961 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:21.817940 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5aa264b-ntw2b" Oct 30 13:40:21.838591 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:21.838566 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42777118-c7bpv"] Oct 30 13:40:21.842464 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:21.842446 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42777118-c7bpv" Oct 30 13:40:21.851730 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:21.851702 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8r4w\" (UniqueName: \"kubernetes.io/projected/ca29257f-a115-4ff7-b977-9f8c488d6db1-kube-api-access-h8r4w\") pod \"maas-api-revoke-sub-a5aa264b-ntw2b\" (UID: \"ca29257f-a115-4ff7-b977-9f8c488d6db1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5aa264b-ntw2b" Oct 30 13:40:21.852057 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:21.851795 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca29257f-a115-4ff7-b977-9f8c488d6db1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5aa264b-ntw2b\" (UID: \"ca29257f-a115-4ff7-b977-9f8c488d6db1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5aa264b-ntw2b" Oct 30 13:40:21.852057 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:21.851854 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca29257f-a115-4ff7-b977-9f8c488d6db1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5aa264b-ntw2b\" (UID: \"ca29257f-a115-4ff7-b977-9f8c488d6db1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5aa264b-ntw2b" Oct 30 13:40:21.852893 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:21.852871 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5aa264b-ntw2b"] Oct 30 13:40:21.858606 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:21.858581 4931 generic.go:356] "Generic (PLEG): container finished" podID="145f3cc9-6c5a-4c88-bc89-da9c956c224d" containerID="f01a91434e52069c23159dc6c2cda66f5810b0cf54b393ffddc4270e0cc0d0ad" exitCode=0 Oct 30 13:40:21.859143 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:21.858664 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de97761d-tf5v5" event={"ID":"145f3cc9-6c5a-4c88-bc89-da9c956c224d","Type":"ContainerDied","Data":"f01a91434e52069c23159dc6c2cda66f5810b0cf54b393ffddc4270e0cc0d0ad"} Oct 30 13:40:21.871746 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:21.871719 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42777118-c7bpv"] Oct 30 13:40:21.953212 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:21.953177 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkbjg\" (UniqueName: \"kubernetes.io/projected/0b18b275-a794-4416-9bbd-81b153e4e98d-kube-api-access-gkbjg\") pod \"maas-api-revoke-sub-42777118-c7bpv\" (UID: \"0b18b275-a794-4416-9bbd-81b153e4e98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42777118-c7bpv" Oct 30 13:40:21.953212 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:21.953227 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b18b275-a794-4416-9bbd-81b153e4e98d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42777118-c7bpv\" (UID: \"0b18b275-a794-4416-9bbd-81b153e4e98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42777118-c7bpv" Oct 30 13:40:21.953507 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:21.953251 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8r4w\" (UniqueName: \"kubernetes.io/projected/ca29257f-a115-4ff7-b977-9f8c488d6db1-kube-api-access-h8r4w\") pod \"maas-api-revoke-sub-a5aa264b-ntw2b\" (UID: \"ca29257f-a115-4ff7-b977-9f8c488d6db1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5aa264b-ntw2b" Oct 30 13:40:21.953573 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:21.953510 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca29257f-a115-4ff7-b977-9f8c488d6db1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5aa264b-ntw2b\" (UID: \"ca29257f-a115-4ff7-b977-9f8c488d6db1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5aa264b-ntw2b" Oct 30 13:40:21.953633 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:21.953577 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca29257f-a115-4ff7-b977-9f8c488d6db1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5aa264b-ntw2b\" (UID: \"ca29257f-a115-4ff7-b977-9f8c488d6db1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5aa264b-ntw2b" Oct 30 13:40:21.953633 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:21.953612 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b18b275-a794-4416-9bbd-81b153e4e98d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42777118-c7bpv\" (UID: \"0b18b275-a794-4416-9bbd-81b153e4e98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42777118-c7bpv" Oct 30 13:40:21.954477 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:21.954449 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca29257f-a115-4ff7-b977-9f8c488d6db1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5aa264b-ntw2b\" (UID: \"ca29257f-a115-4ff7-b977-9f8c488d6db1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5aa264b-ntw2b" Oct 30 13:40:21.955838 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:21.955817 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca29257f-a115-4ff7-b977-9f8c488d6db1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5aa264b-ntw2b\" (UID: \"ca29257f-a115-4ff7-b977-9f8c488d6db1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5aa264b-ntw2b" Oct 30 13:40:21.977813 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:21.977787 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8r4w\" (UniqueName: \"kubernetes.io/projected/ca29257f-a115-4ff7-b977-9f8c488d6db1-kube-api-access-h8r4w\") pod \"maas-api-revoke-sub-a5aa264b-ntw2b\" (UID: \"ca29257f-a115-4ff7-b977-9f8c488d6db1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5aa264b-ntw2b" Oct 30 13:40:22.054998 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:22.054964 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b18b275-a794-4416-9bbd-81b153e4e98d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42777118-c7bpv\" (UID: \"0b18b275-a794-4416-9bbd-81b153e4e98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42777118-c7bpv" Oct 30 13:40:22.055130 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:22.055059 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkbjg\" (UniqueName: \"kubernetes.io/projected/0b18b275-a794-4416-9bbd-81b153e4e98d-kube-api-access-gkbjg\") pod \"maas-api-revoke-sub-42777118-c7bpv\" (UID: \"0b18b275-a794-4416-9bbd-81b153e4e98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42777118-c7bpv" Oct 30 13:40:22.055130 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:22.055087 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b18b275-a794-4416-9bbd-81b153e4e98d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42777118-c7bpv\" (UID: \"0b18b275-a794-4416-9bbd-81b153e4e98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42777118-c7bpv" Oct 30 13:40:22.055751 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:22.055724 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b18b275-a794-4416-9bbd-81b153e4e98d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42777118-c7bpv\" (UID: \"0b18b275-a794-4416-9bbd-81b153e4e98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42777118-c7bpv" Oct 30 13:40:22.057414 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:22.057390 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b18b275-a794-4416-9bbd-81b153e4e98d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42777118-c7bpv\" (UID: \"0b18b275-a794-4416-9bbd-81b153e4e98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42777118-c7bpv" Oct 30 13:40:22.079722 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:22.079667 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkbjg\" (UniqueName: \"kubernetes.io/projected/0b18b275-a794-4416-9bbd-81b153e4e98d-kube-api-access-gkbjg\") pod \"maas-api-revoke-sub-42777118-c7bpv\" (UID: \"0b18b275-a794-4416-9bbd-81b153e4e98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42777118-c7bpv" Oct 30 13:40:22.132238 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:22.132211 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5aa264b-ntw2b" Oct 30 13:40:22.158874 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:22.158846 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42777118-c7bpv" Oct 30 13:40:22.438305 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:22.438276 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5aa264b-ntw2b"] Oct 30 13:40:22.439362 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:40:22.439335 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca29257f_a115_4ff7_b977_9f8c488d6db1.slice/crio-14f8157adb74e795a70a504c55b8ad905b8e35798ac4ef86d7d1572d96b71969 WatchSource:0}: Error finding container 14f8157adb74e795a70a504c55b8ad905b8e35798ac4ef86d7d1572d96b71969: Status 404 returned error can't find the container with id 14f8157adb74e795a70a504c55b8ad905b8e35798ac4ef86d7d1572d96b71969 Oct 30 13:40:22.508232 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:22.508203 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42777118-c7bpv"] Oct 30 13:40:22.508782 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:40:22.508754 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b18b275_a794_4416_9bbd_81b153e4e98d.slice/crio-d2958cef11700262899cdaee35ab580f60b42045da43bd0452cefe05db9e6317 WatchSource:0}: Error finding container d2958cef11700262899cdaee35ab580f60b42045da43bd0452cefe05db9e6317: Status 404 returned error can't find the container with id d2958cef11700262899cdaee35ab580f60b42045da43bd0452cefe05db9e6317 Oct 30 13:40:22.866907 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:22.866871 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42777118-c7bpv" event={"ID":"0b18b275-a794-4416-9bbd-81b153e4e98d","Type":"ContainerStarted","Data":"aa188db15d3972e2f64cc5cc579db8291dba17852c16a8dc84637ec4bad65d5f"} Oct 30 13:40:22.867354 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:22.866932 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42777118-c7bpv" event={"ID":"0b18b275-a794-4416-9bbd-81b153e4e98d","Type":"ContainerStarted","Data":"d2958cef11700262899cdaee35ab580f60b42045da43bd0452cefe05db9e6317"} Oct 30 13:40:22.868745 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:22.868716 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5aa264b-ntw2b" event={"ID":"ca29257f-a115-4ff7-b977-9f8c488d6db1","Type":"ContainerStarted","Data":"ff45ce4d54d46c427a672b760af586d2561fbb8685325031f2f349c4d79128e2"} Oct 30 13:40:22.868857 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:22.868767 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5aa264b-ntw2b" event={"ID":"ca29257f-a115-4ff7-b977-9f8c488d6db1","Type":"ContainerStarted","Data":"14f8157adb74e795a70a504c55b8ad905b8e35798ac4ef86d7d1572d96b71969"} Oct 30 13:40:22.959812 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:22.959747 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5aa264b-ntw2b" podStartSLOduration=1.959732738 podStartE2EDuration="1.959732738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:22.957707852 +0000 UTC m=+3321.561206289" watchObservedRunningTime="2026-10-30 13:40:22.959732738 +0000 UTC m=+3321.563231184" Oct 30 13:40:22.961419 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:22.961379 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42777118-c7bpv" podStartSLOduration=1.961367933 podStartE2EDuration="1.961367933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:22.911481132 +0000 UTC m=+3321.514979577" watchObservedRunningTime="2026-10-30 13:40:22.961367933 +0000 UTC m=+3321.564866373" Oct 30 13:40:23.022162 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:23.022128 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de97761d-tf5v5" Oct 30 13:40:23.068653 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:23.068620 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/145f3cc9-6c5a-4c88-bc89-da9c956c224d-kube-api-access-d4m6f\" (UniqueName: \"kubernetes.io/projected/145f3cc9-6c5a-4c88-bc89-da9c956c224d-kube-api-access-d4m6f\") pod \"145f3cc9-6c5a-4c88-bc89-da9c956c224d\" (UID: \"145f3cc9-6c5a-4c88-bc89-da9c956c224d\") " Oct 30 13:40:23.068981 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:23.068674 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/145f3cc9-6c5a-4c88-bc89-da9c956c224d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/145f3cc9-6c5a-4c88-bc89-da9c956c224d-maas-api-xks-ca\") pod \"145f3cc9-6c5a-4c88-bc89-da9c956c224d\" (UID: \"145f3cc9-6c5a-4c88-bc89-da9c956c224d\") " Oct 30 13:40:23.068981 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:23.068748 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/145f3cc9-6c5a-4c88-bc89-da9c956c224d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/145f3cc9-6c5a-4c88-bc89-da9c956c224d-maas-api-openshift-ca\") pod \"145f3cc9-6c5a-4c88-bc89-da9c956c224d\" (UID: \"145f3cc9-6c5a-4c88-bc89-da9c956c224d\") " Oct 30 13:40:23.069667 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:23.069636 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/145f3cc9-6c5a-4c88-bc89-da9c956c224d-maas-api-openshift-ca" pod "145f3cc9-6c5a-4c88-bc89-da9c956c224d" (UID: "145f3cc9-6c5a-4c88-bc89-da9c956c224d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:40:23.071593 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:23.071564 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/145f3cc9-6c5a-4c88-bc89-da9c956c224d-kube-api-access-d4m6f" pod "145f3cc9-6c5a-4c88-bc89-da9c956c224d" (UID: "145f3cc9-6c5a-4c88-bc89-da9c956c224d"). InnerVolumeSpecName "kube-api-access-d4m6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:23.071806 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:23.071778 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/145f3cc9-6c5a-4c88-bc89-da9c956c224d-maas-api-xks-ca" pod "145f3cc9-6c5a-4c88-bc89-da9c956c224d" (UID: "145f3cc9-6c5a-4c88-bc89-da9c956c224d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:40:23.170995 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:23.170963 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4m6f\" (UniqueName: \"kubernetes.io/projected/145f3cc9-6c5a-4c88-bc89-da9c956c224d-kube-api-access-d4m6f\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:23.170995 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:23.170995 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/145f3cc9-6c5a-4c88-bc89-da9c956c224d-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:23.171183 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:23.171010 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/145f3cc9-6c5a-4c88-bc89-da9c956c224d-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:23.813478 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:23.813376 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" podUID="9577f571-30cd-4043-a3e4-72bf95837552" containerName="main" containerID="cri-o://c3ef6595450b537b0e3c08895c3069621cb7d582458f5ca47f4232bc2d60d04f" gracePeriod=45 Oct 30 13:40:23.827736 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:23.827682 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" podUID="64468005-61ea-4f2a-b72c-9c30c6a4d686" containerName="main" containerID="cri-o://b444cfe4e01b7d9d26ec02fcef11e83b5f216fc92d70f77157d1742768bc21fd" gracePeriod=45 Oct 30 13:40:23.877018 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:23.876983 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de97761d-tf5v5" Oct 30 13:40:23.877468 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:23.876985 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de97761d-tf5v5" event={"ID":"145f3cc9-6c5a-4c88-bc89-da9c956c224d","Type":"ContainerDied","Data":"28fc6d7651e54637407bbad3562e4e0a67be2aad60933552767a48a3ec8892a0"} Oct 30 13:40:23.877468 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:23.877161 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28fc6d7651e54637407bbad3562e4e0a67be2aad60933552767a48a3ec8892a0" Oct 30 13:40:23.878769 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:23.878744 4931 generic.go:356] "Generic (PLEG): container finished" podID="ca29257f-a115-4ff7-b977-9f8c488d6db1" containerID="ff45ce4d54d46c427a672b760af586d2561fbb8685325031f2f349c4d79128e2" exitCode=0 Oct 30 13:40:23.878850 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:23.878817 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5aa264b-ntw2b" event={"ID":"ca29257f-a115-4ff7-b977-9f8c488d6db1","Type":"ContainerDied","Data":"ff45ce4d54d46c427a672b760af586d2561fbb8685325031f2f349c4d79128e2"} Oct 30 13:40:23.880427 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:23.880403 4931 generic.go:356] "Generic (PLEG): container finished" podID="0b18b275-a794-4416-9bbd-81b153e4e98d" containerID="aa188db15d3972e2f64cc5cc579db8291dba17852c16a8dc84637ec4bad65d5f" exitCode=0 Oct 30 13:40:23.880504 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:23.880450 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42777118-c7bpv" event={"ID":"0b18b275-a794-4416-9bbd-81b153e4e98d","Type":"ContainerDied","Data":"aa188db15d3972e2f64cc5cc579db8291dba17852c16a8dc84637ec4bad65d5f"} Oct 30 13:40:23.963501 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:23.963451 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" podUID="9577f571-30cd-4043-a3e4-72bf95837552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": dial tcp 10.132.0.65:8000: connect: connection refused" Oct 30 13:40:24.184870 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.184839 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:40:24.189581 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.189550 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 13:40:24.286071 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.286038 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64468005-61ea-4f2a-b72c-9c30c6a4d686-kube-api-access-7pmp9\" (UniqueName: \"kubernetes.io/projected/64468005-61ea-4f2a-b72c-9c30c6a4d686-kube-api-access-7pmp9\") pod \"64468005-61ea-4f2a-b72c-9c30c6a4d686\" (UID: \"64468005-61ea-4f2a-b72c-9c30c6a4d686\") " Oct 30 13:40:24.286298 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.286096 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64468005-61ea-4f2a-b72c-9c30c6a4d686-home\" (UniqueName: \"kubernetes.io/empty-dir/64468005-61ea-4f2a-b72c-9c30c6a4d686-home\") pod \"64468005-61ea-4f2a-b72c-9c30c6a4d686\" (UID: \"64468005-61ea-4f2a-b72c-9c30c6a4d686\") " Oct 30 13:40:24.286298 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.286159 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9577f571-30cd-4043-a3e4-72bf95837552-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9577f571-30cd-4043-a3e4-72bf95837552-model-cache\") pod \"9577f571-30cd-4043-a3e4-72bf95837552\" (UID: \"9577f571-30cd-4043-a3e4-72bf95837552\") " Oct 30 13:40:24.286298 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.286198 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9577f571-30cd-4043-a3e4-72bf95837552-tls-certs\" (UniqueName: \"kubernetes.io/secret/9577f571-30cd-4043-a3e4-72bf95837552-tls-certs\") pod \"9577f571-30cd-4043-a3e4-72bf95837552\" (UID: \"9577f571-30cd-4043-a3e4-72bf95837552\") " Oct 30 13:40:24.286298 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.286240 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64468005-61ea-4f2a-b72c-9c30c6a4d686-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64468005-61ea-4f2a-b72c-9c30c6a4d686-tmp-dir\") pod \"64468005-61ea-4f2a-b72c-9c30c6a4d686\" (UID: \"64468005-61ea-4f2a-b72c-9c30c6a4d686\") " Oct 30 13:40:24.286298 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.286288 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9577f571-30cd-4043-a3e4-72bf95837552-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9577f571-30cd-4043-a3e4-72bf95837552-tmp-dir\") pod \"9577f571-30cd-4043-a3e4-72bf95837552\" (UID: \"9577f571-30cd-4043-a3e4-72bf95837552\") " Oct 30 13:40:24.286595 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.286317 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9577f571-30cd-4043-a3e4-72bf95837552-kube-api-access-7dtgp\" (UniqueName: \"kubernetes.io/projected/9577f571-30cd-4043-a3e4-72bf95837552-kube-api-access-7dtgp\") pod \"9577f571-30cd-4043-a3e4-72bf95837552\" (UID: \"9577f571-30cd-4043-a3e4-72bf95837552\") " Oct 30 13:40:24.286595 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.286356 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9577f571-30cd-4043-a3e4-72bf95837552-home\" (UniqueName: \"kubernetes.io/empty-dir/9577f571-30cd-4043-a3e4-72bf95837552-home\") pod \"9577f571-30cd-4043-a3e4-72bf95837552\" (UID: \"9577f571-30cd-4043-a3e4-72bf95837552\") " Oct 30 13:40:24.286595 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.286454 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64468005-61ea-4f2a-b72c-9c30c6a4d686-tls-certs\" (UniqueName: \"kubernetes.io/secret/64468005-61ea-4f2a-b72c-9c30c6a4d686-tls-certs\") pod \"64468005-61ea-4f2a-b72c-9c30c6a4d686\" (UID: \"64468005-61ea-4f2a-b72c-9c30c6a4d686\") " Oct 30 13:40:24.286595 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.286457 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64468005-61ea-4f2a-b72c-9c30c6a4d686-home" pod "64468005-61ea-4f2a-b72c-9c30c6a4d686" (UID: "64468005-61ea-4f2a-b72c-9c30c6a4d686"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:24.286595 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.286583 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64468005-61ea-4f2a-b72c-9c30c6a4d686-tmp-dir" pod "64468005-61ea-4f2a-b72c-9c30c6a4d686" (UID: "64468005-61ea-4f2a-b72c-9c30c6a4d686"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:24.286840 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.286634 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64468005-61ea-4f2a-b72c-9c30c6a4d686-dshm\" (UniqueName: \"kubernetes.io/empty-dir/64468005-61ea-4f2a-b72c-9c30c6a4d686-dshm\") pod \"64468005-61ea-4f2a-b72c-9c30c6a4d686\" (UID: \"64468005-61ea-4f2a-b72c-9c30c6a4d686\") " Oct 30 13:40:24.286840 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.286669 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64468005-61ea-4f2a-b72c-9c30c6a4d686-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64468005-61ea-4f2a-b72c-9c30c6a4d686-model-cache\") pod \"64468005-61ea-4f2a-b72c-9c30c6a4d686\" (UID: \"64468005-61ea-4f2a-b72c-9c30c6a4d686\") " Oct 30 13:40:24.286840 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.286715 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9577f571-30cd-4043-a3e4-72bf95837552-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9577f571-30cd-4043-a3e4-72bf95837552-dshm\") pod \"9577f571-30cd-4043-a3e4-72bf95837552\" (UID: \"9577f571-30cd-4043-a3e4-72bf95837552\") " Oct 30 13:40:24.287223 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.287203 4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64468005-61ea-4f2a-b72c-9c30c6a4d686-home\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:24.287381 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.287240 4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64468005-61ea-4f2a-b72c-9c30c6a4d686-tmp-dir\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:24.287511 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.287489 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9577f571-30cd-4043-a3e4-72bf95837552-model-cache" pod "9577f571-30cd-4043-a3e4-72bf95837552" (UID: "9577f571-30cd-4043-a3e4-72bf95837552"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:24.287979 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.287948 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9577f571-30cd-4043-a3e4-72bf95837552-home" pod "9577f571-30cd-4043-a3e4-72bf95837552" (UID: "9577f571-30cd-4043-a3e4-72bf95837552"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:24.288223 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.288151 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9577f571-30cd-4043-a3e4-72bf95837552-tmp-dir" pod "9577f571-30cd-4043-a3e4-72bf95837552" (UID: "9577f571-30cd-4043-a3e4-72bf95837552"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:24.288476 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.288441 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64468005-61ea-4f2a-b72c-9c30c6a4d686-model-cache" pod "64468005-61ea-4f2a-b72c-9c30c6a4d686" (UID: "64468005-61ea-4f2a-b72c-9c30c6a4d686"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:24.288893 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.288866 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64468005-61ea-4f2a-b72c-9c30c6a4d686-kube-api-access-7pmp9" pod "64468005-61ea-4f2a-b72c-9c30c6a4d686" (UID: "64468005-61ea-4f2a-b72c-9c30c6a4d686"). InnerVolumeSpecName "kube-api-access-7pmp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:24.289783 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.289751 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9577f571-30cd-4043-a3e4-72bf95837552-kube-api-access-7dtgp" pod "9577f571-30cd-4043-a3e4-72bf95837552" (UID: "9577f571-30cd-4043-a3e4-72bf95837552"). InnerVolumeSpecName "kube-api-access-7dtgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:24.289894 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.289799 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9577f571-30cd-4043-a3e4-72bf95837552-tls-certs" pod "9577f571-30cd-4043-a3e4-72bf95837552" (UID: "9577f571-30cd-4043-a3e4-72bf95837552"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:40:24.290160 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.290136 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64468005-61ea-4f2a-b72c-9c30c6a4d686-tls-certs" pod "64468005-61ea-4f2a-b72c-9c30c6a4d686" (UID: "64468005-61ea-4f2a-b72c-9c30c6a4d686"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:40:24.290307 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.290190 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9577f571-30cd-4043-a3e4-72bf95837552-dshm" pod "9577f571-30cd-4043-a3e4-72bf95837552" (UID: "9577f571-30cd-4043-a3e4-72bf95837552"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:24.291213 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.291186 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64468005-61ea-4f2a-b72c-9c30c6a4d686-dshm" pod "64468005-61ea-4f2a-b72c-9c30c6a4d686" (UID: "64468005-61ea-4f2a-b72c-9c30c6a4d686"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:24.388565 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.388299 4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64468005-61ea-4f2a-b72c-9c30c6a4d686-tls-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:24.388565 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.388501 4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64468005-61ea-4f2a-b72c-9c30c6a4d686-dshm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:24.388565 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.388548 4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64468005-61ea-4f2a-b72c-9c30c6a4d686-model-cache\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:24.388565 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.388564 4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9577f571-30cd-4043-a3e4-72bf95837552-dshm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:24.388565 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.388577 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pmp9\" (UniqueName: \"kubernetes.io/projected/64468005-61ea-4f2a-b72c-9c30c6a4d686-kube-api-access-7pmp9\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:24.388939 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.388590 4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9577f571-30cd-4043-a3e4-72bf95837552-model-cache\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:24.388939 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.388603 4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9577f571-30cd-4043-a3e4-72bf95837552-tls-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:24.388939 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.388611 4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9577f571-30cd-4043-a3e4-72bf95837552-tmp-dir\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:24.388939 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.388619 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dtgp\" (UniqueName: \"kubernetes.io/projected/9577f571-30cd-4043-a3e4-72bf95837552-kube-api-access-7dtgp\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:24.388939 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.388627 4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9577f571-30cd-4043-a3e4-72bf95837552-home\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:24.869069 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.869033 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:40:24.890199 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.890159 4931 generic.go:356] "Generic (PLEG): container finished" podID="9577f571-30cd-4043-a3e4-72bf95837552" containerID="c3ef6595450b537b0e3c08895c3069621cb7d582458f5ca47f4232bc2d60d04f" exitCode=0 Oct 30 13:40:24.890679 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.890208 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" Oct 30 13:40:24.890679 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.890246 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" event={"ID":"9577f571-30cd-4043-a3e4-72bf95837552","Type":"ContainerDied","Data":"c3ef6595450b537b0e3c08895c3069621cb7d582458f5ca47f4232bc2d60d04f"} Oct 30 13:40:24.890679 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.890330 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk" event={"ID":"9577f571-30cd-4043-a3e4-72bf95837552","Type":"ContainerDied","Data":"4184fd36f69f0deda36442fb4c1a16cfc2be62b79f7b983e63e5ecb70ec0f2b2"} Oct 30 13:40:24.890679 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.890364 4931 scope.go:122] "RemoveContainer" containerID="c3ef6595450b537b0e3c08895c3069621cb7d582458f5ca47f4232bc2d60d04f" Oct 30 13:40:24.891857 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.891829 4931 generic.go:356] "Generic (PLEG): container finished" podID="64468005-61ea-4f2a-b72c-9c30c6a4d686" containerID="b444cfe4e01b7d9d26ec02fcef11e83b5f216fc92d70f77157d1742768bc21fd" exitCode=0 Oct 30 13:40:24.891993 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.891863 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" event={"ID":"64468005-61ea-4f2a-b72c-9c30c6a4d686","Type":"ContainerDied","Data":"b444cfe4e01b7d9d26ec02fcef11e83b5f216fc92d70f77157d1742768bc21fd"} Oct 30 13:40:24.891993 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.891877 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" Oct 30 13:40:24.891993 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.891901 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84" event={"ID":"64468005-61ea-4f2a-b72c-9c30c6a4d686","Type":"ContainerDied","Data":"782a2d893db6bb620cd47450c710e2e31bf94afb9d5989a6a97f1f631396990d"} Oct 30 13:40:24.903130 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.903067 4931 scope.go:122] "RemoveContainer" containerID="c3ef6595450b537b0e3c08895c3069621cb7d582458f5ca47f4232bc2d60d04f" Oct 30 13:40:24.903914 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:40:24.903886 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3ef6595450b537b0e3c08895c3069621cb7d582458f5ca47f4232bc2d60d04f\": container with ID starting with c3ef6595450b537b0e3c08895c3069621cb7d582458f5ca47f4232bc2d60d04f not found: ID does not exist" containerID="c3ef6595450b537b0e3c08895c3069621cb7d582458f5ca47f4232bc2d60d04f" Oct 30 13:40:24.903994 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.903924 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3ef6595450b537b0e3c08895c3069621cb7d582458f5ca47f4232bc2d60d04f"} err="failed to get container status \"c3ef6595450b537b0e3c08895c3069621cb7d582458f5ca47f4232bc2d60d04f\": rpc error: code = NotFound desc = could not find container \"c3ef6595450b537b0e3c08895c3069621cb7d582458f5ca47f4232bc2d60d04f\": container with ID starting with c3ef6595450b537b0e3c08895c3069621cb7d582458f5ca47f4232bc2d60d04f not found: ID does not exist" Oct 30 13:40:24.903994 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.903950 4931 scope.go:122] "RemoveContainer" containerID="b444cfe4e01b7d9d26ec02fcef11e83b5f216fc92d70f77157d1742768bc21fd" Oct 30 13:40:24.914727 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.914698 4931 scope.go:122] "RemoveContainer" containerID="b444cfe4e01b7d9d26ec02fcef11e83b5f216fc92d70f77157d1742768bc21fd" Oct 30 13:40:24.915044 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:40:24.915023 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b444cfe4e01b7d9d26ec02fcef11e83b5f216fc92d70f77157d1742768bc21fd\": container with ID starting with b444cfe4e01b7d9d26ec02fcef11e83b5f216fc92d70f77157d1742768bc21fd not found: ID does not exist" containerID="b444cfe4e01b7d9d26ec02fcef11e83b5f216fc92d70f77157d1742768bc21fd" Oct 30 13:40:24.915098 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.915061 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b444cfe4e01b7d9d26ec02fcef11e83b5f216fc92d70f77157d1742768bc21fd"} err="failed to get container status \"b444cfe4e01b7d9d26ec02fcef11e83b5f216fc92d70f77157d1742768bc21fd\": rpc error: code = NotFound desc = could not find container \"b444cfe4e01b7d9d26ec02fcef11e83b5f216fc92d70f77157d1742768bc21fd\": container with ID starting with b444cfe4e01b7d9d26ec02fcef11e83b5f216fc92d70f77157d1742768bc21fd not found: ID does not exist" Oct 30 13:40:24.956027 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.955990 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk"] Oct 30 13:40:24.962495 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:24.962463 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-8b46c9/premium-w1-8b46c9-kserve-bf4bf4f8f-xc8sk"] Oct 30 13:40:25.004831 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:25.003190 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84"] Oct 30 13:40:25.008359 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:25.008327 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-8b46c9/simulator-w1-8b46c9-kserve-7d57d6f588-ppb84"] Oct 30 13:40:25.101591 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:25.101558 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42777118-c7bpv" Oct 30 13:40:25.103878 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:25.103859 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5aa264b-ntw2b" Oct 30 13:40:25.210585 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:25.210540 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca29257f-a115-4ff7-b977-9f8c488d6db1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca29257f-a115-4ff7-b977-9f8c488d6db1-maas-api-xks-ca\") pod \"ca29257f-a115-4ff7-b977-9f8c488d6db1\" (UID: \"ca29257f-a115-4ff7-b977-9f8c488d6db1\") " Oct 30 13:40:25.210758 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:25.210637 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b18b275-a794-4416-9bbd-81b153e4e98d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b18b275-a794-4416-9bbd-81b153e4e98d-maas-api-xks-ca\") pod \"0b18b275-a794-4416-9bbd-81b153e4e98d\" (UID: \"0b18b275-a794-4416-9bbd-81b153e4e98d\") " Oct 30 13:40:25.210758 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:25.210697 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca29257f-a115-4ff7-b977-9f8c488d6db1-kube-api-access-h8r4w\" (UniqueName: \"kubernetes.io/projected/ca29257f-a115-4ff7-b977-9f8c488d6db1-kube-api-access-h8r4w\") pod \"ca29257f-a115-4ff7-b977-9f8c488d6db1\" (UID: \"ca29257f-a115-4ff7-b977-9f8c488d6db1\") " Oct 30 13:40:25.210758 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:25.210746 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b18b275-a794-4416-9bbd-81b153e4e98d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b18b275-a794-4416-9bbd-81b153e4e98d-maas-api-openshift-ca\") pod \"0b18b275-a794-4416-9bbd-81b153e4e98d\" (UID: \"0b18b275-a794-4416-9bbd-81b153e4e98d\") " Oct 30 13:40:25.210984 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:25.210963 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b18b275-a794-4416-9bbd-81b153e4e98d-kube-api-access-gkbjg\" (UniqueName: \"kubernetes.io/projected/0b18b275-a794-4416-9bbd-81b153e4e98d-kube-api-access-gkbjg\") pod \"0b18b275-a794-4416-9bbd-81b153e4e98d\" (UID: \"0b18b275-a794-4416-9bbd-81b153e4e98d\") " Oct 30 13:40:25.211093 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:25.211001 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca29257f-a115-4ff7-b977-9f8c488d6db1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca29257f-a115-4ff7-b977-9f8c488d6db1-maas-api-openshift-ca\") pod \"ca29257f-a115-4ff7-b977-9f8c488d6db1\" (UID: \"ca29257f-a115-4ff7-b977-9f8c488d6db1\") " Oct 30 13:40:25.211198 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:25.211172 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b18b275-a794-4416-9bbd-81b153e4e98d-maas-api-openshift-ca" pod "0b18b275-a794-4416-9bbd-81b153e4e98d" (UID: "0b18b275-a794-4416-9bbd-81b153e4e98d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:40:25.211557 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:25.211529 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca29257f-a115-4ff7-b977-9f8c488d6db1-maas-api-openshift-ca" pod "ca29257f-a115-4ff7-b977-9f8c488d6db1" (UID: "ca29257f-a115-4ff7-b977-9f8c488d6db1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:40:25.211751 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:25.211732 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca29257f-a115-4ff7-b977-9f8c488d6db1-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:25.211829 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:25.211780 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b18b275-a794-4416-9bbd-81b153e4e98d-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:25.213400 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:25.213367 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b18b275-a794-4416-9bbd-81b153e4e98d-kube-api-access-gkbjg" pod "0b18b275-a794-4416-9bbd-81b153e4e98d" (UID: "0b18b275-a794-4416-9bbd-81b153e4e98d"). InnerVolumeSpecName "kube-api-access-gkbjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:25.213934 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:25.213906 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca29257f-a115-4ff7-b977-9f8c488d6db1-maas-api-xks-ca" pod "ca29257f-a115-4ff7-b977-9f8c488d6db1" (UID: "ca29257f-a115-4ff7-b977-9f8c488d6db1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:40:25.213934 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:25.213916 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b18b275-a794-4416-9bbd-81b153e4e98d-maas-api-xks-ca" pod "0b18b275-a794-4416-9bbd-81b153e4e98d" (UID: "0b18b275-a794-4416-9bbd-81b153e4e98d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:40:25.214070 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:25.213965 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca29257f-a115-4ff7-b977-9f8c488d6db1-kube-api-access-h8r4w" pod "ca29257f-a115-4ff7-b977-9f8c488d6db1" (UID: "ca29257f-a115-4ff7-b977-9f8c488d6db1"). InnerVolumeSpecName "kube-api-access-h8r4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:25.312770 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:25.312733 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca29257f-a115-4ff7-b977-9f8c488d6db1-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:25.312770 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:25.312766 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b18b275-a794-4416-9bbd-81b153e4e98d-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:25.312965 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:25.312781 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8r4w\" (UniqueName: \"kubernetes.io/projected/ca29257f-a115-4ff7-b977-9f8c488d6db1-kube-api-access-h8r4w\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:25.312965 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:25.312794 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkbjg\" (UniqueName: \"kubernetes.io/projected/0b18b275-a794-4416-9bbd-81b153e4e98d-kube-api-access-gkbjg\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:25.512911 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:25.512825 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64468005-61ea-4f2a-b72c-9c30c6a4d686" path="/var/lib/kubelet/pods/64468005-61ea-4f2a-b72c-9c30c6a4d686/volumes" Oct 30 13:40:25.513428 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:25.513405 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9577f571-30cd-4043-a3e4-72bf95837552" path="/var/lib/kubelet/pods/9577f571-30cd-4043-a3e4-72bf95837552/volumes" Oct 30 13:40:25.905576 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:25.905494 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5aa264b-ntw2b" Oct 30 13:40:25.905576 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:25.905525 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5aa264b-ntw2b" event={"ID":"ca29257f-a115-4ff7-b977-9f8c488d6db1","Type":"ContainerDied","Data":"14f8157adb74e795a70a504c55b8ad905b8e35798ac4ef86d7d1572d96b71969"} Oct 30 13:40:25.906088 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:25.905583 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14f8157adb74e795a70a504c55b8ad905b8e35798ac4ef86d7d1572d96b71969" Oct 30 13:40:25.907080 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:25.907056 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42777118-c7bpv" Oct 30 13:40:25.907080 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:25.907075 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42777118-c7bpv" event={"ID":"0b18b275-a794-4416-9bbd-81b153e4e98d","Type":"ContainerDied","Data":"d2958cef11700262899cdaee35ab580f60b42045da43bd0452cefe05db9e6317"} Oct 30 13:40:25.907308 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:25.907115 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2958cef11700262899cdaee35ab580f60b42045da43bd0452cefe05db9e6317" Oct 30 13:40:26.520434 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:26.520387 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" podUID="0c5a9f00-6d5b-4860-a210-2fec24afb322" containerName="main" containerID="cri-o://42316409e0da023240a8e21ef23c9ddc169a8cc3c18e57e2d1cfbb26d722b4bc" gracePeriod=45 Oct 30 13:40:26.572501 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:26.572460 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" podUID="d345fdd5-407b-4162-a2cd-64a40068cb3c" containerName="main" containerID="cri-o://a9dc518c33e4978fb35eaf3bc23a73e29be95c7b9e8d6b23b339a665eb776fcf" gracePeriod=45 Oct 30 13:40:26.927172 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:26.927134 4931 generic.go:356] "Generic (PLEG): container finished" podID="d345fdd5-407b-4162-a2cd-64a40068cb3c" containerID="a9dc518c33e4978fb35eaf3bc23a73e29be95c7b9e8d6b23b339a665eb776fcf" exitCode=0 Oct 30 13:40:26.927768 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:26.927328 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" event={"ID":"d345fdd5-407b-4162-a2cd-64a40068cb3c","Type":"ContainerDied","Data":"a9dc518c33e4978fb35eaf3bc23a73e29be95c7b9e8d6b23b339a665eb776fcf"} Oct 30 13:40:26.929706 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:26.929492 4931 generic.go:356] "Generic (PLEG): container finished" podID="0c5a9f00-6d5b-4860-a210-2fec24afb322" containerID="42316409e0da023240a8e21ef23c9ddc169a8cc3c18e57e2d1cfbb26d722b4bc" exitCode=0 Oct 30 13:40:26.929706 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:26.929622 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" event={"ID":"0c5a9f00-6d5b-4860-a210-2fec24afb322","Type":"ContainerDied","Data":"42316409e0da023240a8e21ef23c9ddc169a8cc3c18e57e2d1cfbb26d722b4bc"} Oct 30 13:40:26.988860 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:26.988830 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 13:40:27.047675 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.046027 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c5a9f00-6d5b-4860-a210-2fec24afb322-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c5a9f00-6d5b-4860-a210-2fec24afb322-dshm\") pod \"0c5a9f00-6d5b-4860-a210-2fec24afb322\" (UID: \"0c5a9f00-6d5b-4860-a210-2fec24afb322\") " Oct 30 13:40:27.047675 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.046083 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c5a9f00-6d5b-4860-a210-2fec24afb322-tls-certs\" (UniqueName: \"kubernetes.io/secret/0c5a9f00-6d5b-4860-a210-2fec24afb322-tls-certs\") pod \"0c5a9f00-6d5b-4860-a210-2fec24afb322\" (UID: \"0c5a9f00-6d5b-4860-a210-2fec24afb322\") " Oct 30 13:40:27.047675 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.046165 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c5a9f00-6d5b-4860-a210-2fec24afb322-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c5a9f00-6d5b-4860-a210-2fec24afb322-model-cache\") pod \"0c5a9f00-6d5b-4860-a210-2fec24afb322\" (UID: \"0c5a9f00-6d5b-4860-a210-2fec24afb322\") " Oct 30 13:40:27.047675 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.046217 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c5a9f00-6d5b-4860-a210-2fec24afb322-kube-api-access-f28bn\" (UniqueName: \"kubernetes.io/projected/0c5a9f00-6d5b-4860-a210-2fec24afb322-kube-api-access-f28bn\") pod \"0c5a9f00-6d5b-4860-a210-2fec24afb322\" (UID: \"0c5a9f00-6d5b-4860-a210-2fec24afb322\") " Oct 30 13:40:27.047675 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.046253 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c5a9f00-6d5b-4860-a210-2fec24afb322-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c5a9f00-6d5b-4860-a210-2fec24afb322-tmp-dir\") pod \"0c5a9f00-6d5b-4860-a210-2fec24afb322\" (UID: \"0c5a9f00-6d5b-4860-a210-2fec24afb322\") " Oct 30 13:40:27.047675 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.046338 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c5a9f00-6d5b-4860-a210-2fec24afb322-home\" (UniqueName: \"kubernetes.io/empty-dir/0c5a9f00-6d5b-4860-a210-2fec24afb322-home\") pod \"0c5a9f00-6d5b-4860-a210-2fec24afb322\" (UID: \"0c5a9f00-6d5b-4860-a210-2fec24afb322\") " Oct 30 13:40:27.050936 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.050900 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c5a9f00-6d5b-4860-a210-2fec24afb322-home" pod "0c5a9f00-6d5b-4860-a210-2fec24afb322" (UID: "0c5a9f00-6d5b-4860-a210-2fec24afb322"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:27.053910 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.053863 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c5a9f00-6d5b-4860-a210-2fec24afb322-tmp-dir" pod "0c5a9f00-6d5b-4860-a210-2fec24afb322" (UID: "0c5a9f00-6d5b-4860-a210-2fec24afb322"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:27.061410 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.061372 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c5a9f00-6d5b-4860-a210-2fec24afb322-model-cache" pod "0c5a9f00-6d5b-4860-a210-2fec24afb322" (UID: "0c5a9f00-6d5b-4860-a210-2fec24afb322"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:27.061686 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.061663 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c5a9f00-6d5b-4860-a210-2fec24afb322-dshm" pod "0c5a9f00-6d5b-4860-a210-2fec24afb322" (UID: "0c5a9f00-6d5b-4860-a210-2fec24afb322"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:27.062674 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.062642 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c5a9f00-6d5b-4860-a210-2fec24afb322-tls-certs" pod "0c5a9f00-6d5b-4860-a210-2fec24afb322" (UID: "0c5a9f00-6d5b-4860-a210-2fec24afb322"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:40:27.065329 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.065019 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 13:40:27.066448 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.066405 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c5a9f00-6d5b-4860-a210-2fec24afb322-kube-api-access-f28bn" pod "0c5a9f00-6d5b-4860-a210-2fec24afb322" (UID: "0c5a9f00-6d5b-4860-a210-2fec24afb322"). InnerVolumeSpecName "kube-api-access-f28bn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:27.148134 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.148039 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d345fdd5-407b-4162-a2cd-64a40068cb3c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d345fdd5-407b-4162-a2cd-64a40068cb3c-tmp-dir\") pod \"d345fdd5-407b-4162-a2cd-64a40068cb3c\" (UID: \"d345fdd5-407b-4162-a2cd-64a40068cb3c\") " Oct 30 13:40:27.148134 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.148125 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d345fdd5-407b-4162-a2cd-64a40068cb3c-tls-certs\" (UniqueName: \"kubernetes.io/secret/d345fdd5-407b-4162-a2cd-64a40068cb3c-tls-certs\") pod \"d345fdd5-407b-4162-a2cd-64a40068cb3c\" (UID: \"d345fdd5-407b-4162-a2cd-64a40068cb3c\") " Oct 30 13:40:27.148452 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.148153 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d345fdd5-407b-4162-a2cd-64a40068cb3c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d345fdd5-407b-4162-a2cd-64a40068cb3c-dshm\") pod \"d345fdd5-407b-4162-a2cd-64a40068cb3c\" (UID: \"d345fdd5-407b-4162-a2cd-64a40068cb3c\") " Oct 30 13:40:27.148452 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.148254 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d345fdd5-407b-4162-a2cd-64a40068cb3c-home\" (UniqueName: \"kubernetes.io/empty-dir/d345fdd5-407b-4162-a2cd-64a40068cb3c-home\") pod \"d345fdd5-407b-4162-a2cd-64a40068cb3c\" (UID: \"d345fdd5-407b-4162-a2cd-64a40068cb3c\") " Oct 30 13:40:27.148452 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.148309 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d345fdd5-407b-4162-a2cd-64a40068cb3c-kube-api-access-fq79j\" (UniqueName: \"kubernetes.io/projected/d345fdd5-407b-4162-a2cd-64a40068cb3c-kube-api-access-fq79j\") pod \"d345fdd5-407b-4162-a2cd-64a40068cb3c\" (UID: \"d345fdd5-407b-4162-a2cd-64a40068cb3c\") " Oct 30 13:40:27.148452 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.148349 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d345fdd5-407b-4162-a2cd-64a40068cb3c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d345fdd5-407b-4162-a2cd-64a40068cb3c-model-cache\") pod \"d345fdd5-407b-4162-a2cd-64a40068cb3c\" (UID: \"d345fdd5-407b-4162-a2cd-64a40068cb3c\") " Oct 30 13:40:27.149542 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.148652 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d345fdd5-407b-4162-a2cd-64a40068cb3c-model-cache" pod "d345fdd5-407b-4162-a2cd-64a40068cb3c" (UID: "d345fdd5-407b-4162-a2cd-64a40068cb3c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:27.149542 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.148879 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d345fdd5-407b-4162-a2cd-64a40068cb3c-tmp-dir" pod "d345fdd5-407b-4162-a2cd-64a40068cb3c" (UID: "d345fdd5-407b-4162-a2cd-64a40068cb3c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:27.149542 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.149422 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d345fdd5-407b-4162-a2cd-64a40068cb3c-home" pod "d345fdd5-407b-4162-a2cd-64a40068cb3c" (UID: "d345fdd5-407b-4162-a2cd-64a40068cb3c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:27.149788 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.149719 4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c5a9f00-6d5b-4860-a210-2fec24afb322-model-cache\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:27.149788 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.149751 4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d345fdd5-407b-4162-a2cd-64a40068cb3c-home\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:27.149788 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.149766 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f28bn\" (UniqueName: \"kubernetes.io/projected/0c5a9f00-6d5b-4860-a210-2fec24afb322-kube-api-access-f28bn\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:27.149788 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.149782 4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c5a9f00-6d5b-4860-a210-2fec24afb322-tmp-dir\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:27.150018 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.149798 4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d345fdd5-407b-4162-a2cd-64a40068cb3c-model-cache\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:27.150018 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.149810 4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c5a9f00-6d5b-4860-a210-2fec24afb322-home\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:27.150018 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.149823 4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d345fdd5-407b-4162-a2cd-64a40068cb3c-tmp-dir\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:27.150018 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.149836 4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c5a9f00-6d5b-4860-a210-2fec24afb322-dshm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:27.150018 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.149848 4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c5a9f00-6d5b-4860-a210-2fec24afb322-tls-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:27.152254 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.152224 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d345fdd5-407b-4162-a2cd-64a40068cb3c-kube-api-access-fq79j" pod "d345fdd5-407b-4162-a2cd-64a40068cb3c" (UID: "d345fdd5-407b-4162-a2cd-64a40068cb3c"). InnerVolumeSpecName "kube-api-access-fq79j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:27.161969 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.161943 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d345fdd5-407b-4162-a2cd-64a40068cb3c-tls-certs" pod "d345fdd5-407b-4162-a2cd-64a40068cb3c" (UID: "d345fdd5-407b-4162-a2cd-64a40068cb3c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:40:27.164406 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.164380 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d345fdd5-407b-4162-a2cd-64a40068cb3c-dshm" pod "d345fdd5-407b-4162-a2cd-64a40068cb3c" (UID: "d345fdd5-407b-4162-a2cd-64a40068cb3c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:27.251597 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.251557 4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d345fdd5-407b-4162-a2cd-64a40068cb3c-tls-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:27.251597 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.251597 4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d345fdd5-407b-4162-a2cd-64a40068cb3c-dshm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:27.251898 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.251611 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq79j\" (UniqueName: \"kubernetes.io/projected/d345fdd5-407b-4162-a2cd-64a40068cb3c-kube-api-access-fq79j\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:27.938547 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.938451 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" Oct 30 13:40:27.938547 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.938481 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c" event={"ID":"d345fdd5-407b-4162-a2cd-64a40068cb3c","Type":"ContainerDied","Data":"32ce3755ba62b725d915ac447d981316c121527a0d146509aa338c18ce849bc5"} Oct 30 13:40:27.938547 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.938529 4931 scope.go:122] "RemoveContainer" containerID="a9dc518c33e4978fb35eaf3bc23a73e29be95c7b9e8d6b23b339a665eb776fcf" Oct 30 13:40:27.940339 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.940181 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" Oct 30 13:40:27.940339 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.940215 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf" event={"ID":"0c5a9f00-6d5b-4860-a210-2fec24afb322","Type":"ContainerDied","Data":"a33a7de8fe933f94afc351ca53c207dfa49a6588d862ac0f2a66484e23aebd66"} Oct 30 13:40:27.949706 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.949687 4931 scope.go:122] "RemoveContainer" containerID="42316409e0da023240a8e21ef23c9ddc169a8cc3c18e57e2d1cfbb26d722b4bc" Oct 30 13:40:27.997636 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:27.997608 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf"] Oct 30 13:40:28.010694 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:28.010669 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c90b9e/premium-w0-c90b9e-kserve-5cdc445666-4n5lf"] Oct 30 13:40:28.036300 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:28.036245 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c"] Oct 30 13:40:28.045039 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:28.044989 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c90b9e/simulator-w0-c90b9e-kserve-798ddd7fcf-lk57c"] Oct 30 13:40:29.513819 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:29.513782 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c5a9f00-6d5b-4860-a210-2fec24afb322" path="/var/lib/kubelet/pods/0c5a9f00-6d5b-4860-a210-2fec24afb322/volumes" Oct 30 13:40:29.514393 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:29.514336 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d345fdd5-407b-4162-a2cd-64a40068cb3c" path="/var/lib/kubelet/pods/d345fdd5-407b-4162-a2cd-64a40068cb3c/volumes" Oct 30 13:40:30.352832 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:30.352792 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd"] Oct 30 13:40:30.355343 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:30.353067 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" podUID="376b917f-91f0-4239-a1b3-eac12620d76e" containerName="istio-proxy" containerID="cri-o://03d514035c1e421392564bd70f0dcbed3806b16e4f6cf47d648d4c04a3b1df06" gracePeriod=30 Oct 30 13:40:31.047054 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.047018 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd90cb-bzxnl"] Oct 30 13:40:31.047528 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.047514 4931 cpu_manager.go:414] "containerMap: removing container" podUID="0b18b275-a794-4416-9bbd-81b153e4e98d" containerName="revoke-keys" Oct 30 13:40:31.047576 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.047529 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b18b275-a794-4416-9bbd-81b153e4e98d" containerName="revoke-keys" Oct 30 13:40:31.047576 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.047549 4931 cpu_manager.go:414] "containerMap: removing container" podUID="0c5a9f00-6d5b-4860-a210-2fec24afb322" containerName="main" Oct 30 13:40:31.047576 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.047554 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c5a9f00-6d5b-4860-a210-2fec24afb322" containerName="main" Oct 30 13:40:31.047576 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.047561 4931 cpu_manager.go:414] "containerMap: removing container" podUID="ca29257f-a115-4ff7-b977-9f8c488d6db1" containerName="revoke-keys" Oct 30 13:40:31.047576 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.047566 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca29257f-a115-4ff7-b977-9f8c488d6db1" containerName="revoke-keys" Oct 30 13:40:31.047758 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.047580 4931 cpu_manager.go:414] "containerMap: removing container" podUID="64468005-61ea-4f2a-b72c-9c30c6a4d686" containerName="main" Oct 30 13:40:31.047758 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.047586 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64468005-61ea-4f2a-b72c-9c30c6a4d686" containerName="main" Oct 30 13:40:31.047758 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.047600 4931 cpu_manager.go:414] "containerMap: removing container" podUID="d345fdd5-407b-4162-a2cd-64a40068cb3c" containerName="main" Oct 30 13:40:31.047758 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.047604 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d345fdd5-407b-4162-a2cd-64a40068cb3c" containerName="main" Oct 30 13:40:31.047758 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.047611 4931 cpu_manager.go:414] "containerMap: removing container" podUID="145f3cc9-6c5a-4c88-bc89-da9c956c224d" containerName="revoke-keys" Oct 30 13:40:31.047758 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.047615 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="145f3cc9-6c5a-4c88-bc89-da9c956c224d" containerName="revoke-keys" Oct 30 13:40:31.047758 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.047627 4931 cpu_manager.go:414] "containerMap: removing container" podUID="9577f571-30cd-4043-a3e4-72bf95837552" containerName="main" Oct 30 13:40:31.047758 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.047631 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9577f571-30cd-4043-a3e4-72bf95837552" containerName="main" Oct 30 13:40:31.047758 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.047704 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="64468005-61ea-4f2a-b72c-9c30c6a4d686" containerName="main" Oct 30 13:40:31.047758 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.047713 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="d345fdd5-407b-4162-a2cd-64a40068cb3c" containerName="main" Oct 30 13:40:31.047758 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.047721 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="145f3cc9-6c5a-4c88-bc89-da9c956c224d" containerName="revoke-keys" Oct 30 13:40:31.047758 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.047729 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="9577f571-30cd-4043-a3e4-72bf95837552" containerName="main" Oct 30 13:40:31.047758 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.047736 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b18b275-a794-4416-9bbd-81b153e4e98d" containerName="revoke-keys" Oct 30 13:40:31.047758 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.047743 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c5a9f00-6d5b-4860-a210-2fec24afb322" containerName="main" Oct 30 13:40:31.047758 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.047749 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca29257f-a115-4ff7-b977-9f8c488d6db1" containerName="revoke-keys" Oct 30 13:40:31.057343 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.056258 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd90cb-bzxnl" Oct 30 13:40:31.072358 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.072327 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd90cb-bzxnl"] Oct 30 13:40:31.086109 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.086079 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:40:31.086557 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.086534 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:40:31.205833 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.205791 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9d97\" (UniqueName: \"kubernetes.io/projected/a00926d5-b69b-4a8a-b076-68ad63208f30-kube-api-access-n9d97\") pod \"maas-api-revoke-keys-e2e-worker-w6-dd90cb-bzxnl\" (UID: \"a00926d5-b69b-4a8a-b076-68ad63208f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd90cb-bzxnl" Oct 30 13:40:31.206043 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.205923 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a00926d5-b69b-4a8a-b076-68ad63208f30-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-dd90cb-bzxnl\" (UID: \"a00926d5-b69b-4a8a-b076-68ad63208f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd90cb-bzxnl" Oct 30 13:40:31.206043 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.206005 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a00926d5-b69b-4a8a-b076-68ad63208f30-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-dd90cb-bzxnl\" (UID: \"a00926d5-b69b-4a8a-b076-68ad63208f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd90cb-bzxnl" Oct 30 13:40:31.307625 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.307540 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9d97\" (UniqueName: \"kubernetes.io/projected/a00926d5-b69b-4a8a-b076-68ad63208f30-kube-api-access-n9d97\") pod \"maas-api-revoke-keys-e2e-worker-w6-dd90cb-bzxnl\" (UID: \"a00926d5-b69b-4a8a-b076-68ad63208f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd90cb-bzxnl" Oct 30 13:40:31.307791 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.307634 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a00926d5-b69b-4a8a-b076-68ad63208f30-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-dd90cb-bzxnl\" (UID: \"a00926d5-b69b-4a8a-b076-68ad63208f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd90cb-bzxnl" Oct 30 13:40:31.307791 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.307725 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a00926d5-b69b-4a8a-b076-68ad63208f30-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-dd90cb-bzxnl\" (UID: \"a00926d5-b69b-4a8a-b076-68ad63208f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd90cb-bzxnl" Oct 30 13:40:31.308341 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.308315 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a00926d5-b69b-4a8a-b076-68ad63208f30-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-dd90cb-bzxnl\" (UID: \"a00926d5-b69b-4a8a-b076-68ad63208f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd90cb-bzxnl" Oct 30 13:40:31.310077 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.310057 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a00926d5-b69b-4a8a-b076-68ad63208f30-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-dd90cb-bzxnl\" (UID: \"a00926d5-b69b-4a8a-b076-68ad63208f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd90cb-bzxnl" Oct 30 13:40:31.332722 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.332694 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9d97\" (UniqueName: \"kubernetes.io/projected/a00926d5-b69b-4a8a-b076-68ad63208f30-kube-api-access-n9d97\") pod \"maas-api-revoke-keys-e2e-worker-w6-dd90cb-bzxnl\" (UID: \"a00926d5-b69b-4a8a-b076-68ad63208f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd90cb-bzxnl" Oct 30 13:40:31.373000 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.372963 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd90cb-bzxnl" Oct 30 13:40:31.718988 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.718959 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd90cb-bzxnl"] Oct 30 13:40:31.719685 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:40:31.719655 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda00926d5_b69b_4a8a_b076_68ad63208f30.slice/crio-4eb9cda4569203f6b2789cae20337cdb3c3974fb4991c2f4266c9d44b8645480 WatchSource:0}: Error finding container 4eb9cda4569203f6b2789cae20337cdb3c3974fb4991c2f4266c9d44b8645480: Status 404 returned error can't find the container with id 4eb9cda4569203f6b2789cae20337cdb3c3974fb4991c2f4266c9d44b8645480 Oct 30 13:40:31.973707 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:31.973636 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd90cb-bzxnl" event={"ID":"a00926d5-b69b-4a8a-b076-68ad63208f30","Type":"ContainerStarted","Data":"4eb9cda4569203f6b2789cae20337cdb3c3974fb4991c2f4266c9d44b8645480"} Oct 30 13:40:32.981522 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:32.981488 4931 generic.go:356] "Generic (PLEG): container finished" podID="a00926d5-b69b-4a8a-b076-68ad63208f30" containerID="6c0ce9d0cfdbf295cb012f5a343eb19f634e1fb710b08a9388906920ca51ed4d" exitCode=0 Oct 30 13:40:32.981997 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:32.981557 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd90cb-bzxnl" event={"ID":"a00926d5-b69b-4a8a-b076-68ad63208f30","Type":"ContainerDied","Data":"6c0ce9d0cfdbf295cb012f5a343eb19f634e1fb710b08a9388906920ca51ed4d"} Oct 30 13:40:34.133017 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:34.132993 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd90cb-bzxnl" Oct 30 13:40:34.244491 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:34.244407 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a00926d5-b69b-4a8a-b076-68ad63208f30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a00926d5-b69b-4a8a-b076-68ad63208f30-maas-api-openshift-ca\") pod \"a00926d5-b69b-4a8a-b076-68ad63208f30\" (UID: \"a00926d5-b69b-4a8a-b076-68ad63208f30\") " Oct 30 13:40:34.244665 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:34.244539 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a00926d5-b69b-4a8a-b076-68ad63208f30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a00926d5-b69b-4a8a-b076-68ad63208f30-maas-api-xks-ca\") pod \"a00926d5-b69b-4a8a-b076-68ad63208f30\" (UID: \"a00926d5-b69b-4a8a-b076-68ad63208f30\") " Oct 30 13:40:34.244665 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:34.244563 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a00926d5-b69b-4a8a-b076-68ad63208f30-kube-api-access-n9d97\" (UniqueName: \"kubernetes.io/projected/a00926d5-b69b-4a8a-b076-68ad63208f30-kube-api-access-n9d97\") pod \"a00926d5-b69b-4a8a-b076-68ad63208f30\" (UID: \"a00926d5-b69b-4a8a-b076-68ad63208f30\") " Oct 30 13:40:34.244801 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:34.244748 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a00926d5-b69b-4a8a-b076-68ad63208f30-maas-api-openshift-ca" pod "a00926d5-b69b-4a8a-b076-68ad63208f30" (UID: "a00926d5-b69b-4a8a-b076-68ad63208f30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:40:34.244921 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:34.244905 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a00926d5-b69b-4a8a-b076-68ad63208f30-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:34.246697 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:34.246671 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a00926d5-b69b-4a8a-b076-68ad63208f30-maas-api-xks-ca" pod "a00926d5-b69b-4a8a-b076-68ad63208f30" (UID: "a00926d5-b69b-4a8a-b076-68ad63208f30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:40:34.246816 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:34.246770 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a00926d5-b69b-4a8a-b076-68ad63208f30-kube-api-access-n9d97" pod "a00926d5-b69b-4a8a-b076-68ad63208f30" (UID: "a00926d5-b69b-4a8a-b076-68ad63208f30"). InnerVolumeSpecName "kube-api-access-n9d97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:34.346149 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:34.346117 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a00926d5-b69b-4a8a-b076-68ad63208f30-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:34.346149 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:34.346145 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9d97\" (UniqueName: \"kubernetes.io/projected/a00926d5-b69b-4a8a-b076-68ad63208f30-kube-api-access-n9d97\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:34.998395 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:34.998357 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd90cb-bzxnl" event={"ID":"a00926d5-b69b-4a8a-b076-68ad63208f30","Type":"ContainerDied","Data":"4eb9cda4569203f6b2789cae20337cdb3c3974fb4991c2f4266c9d44b8645480"} Oct 30 13:40:34.998395 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:34.998390 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd90cb-bzxnl" Oct 30 13:40:34.998623 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:34.998443 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4eb9cda4569203f6b2789cae20337cdb3c3974fb4991c2f4266c9d44b8645480" Oct 30 13:40:35.672811 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:35.672770 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:40:35.761619 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:35.761585 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/376b917f-91f0-4239-a1b3-eac12620d76e-istio-token\" (UniqueName: \"kubernetes.io/projected/376b917f-91f0-4239-a1b3-eac12620d76e-istio-token\") pod \"376b917f-91f0-4239-a1b3-eac12620d76e\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " Oct 30 13:40:35.761787 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:35.761716 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/376b917f-91f0-4239-a1b3-eac12620d76e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/376b917f-91f0-4239-a1b3-eac12620d76e-credential-socket\") pod \"376b917f-91f0-4239-a1b3-eac12620d76e\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " Oct 30 13:40:35.761787 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:35.761740 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/376b917f-91f0-4239-a1b3-eac12620d76e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/376b917f-91f0-4239-a1b3-eac12620d76e-istio-podinfo\") pod \"376b917f-91f0-4239-a1b3-eac12620d76e\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " Oct 30 13:40:35.761787 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:35.761772 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/376b917f-91f0-4239-a1b3-eac12620d76e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/376b917f-91f0-4239-a1b3-eac12620d76e-istio-data\") pod \"376b917f-91f0-4239-a1b3-eac12620d76e\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " Oct 30 13:40:35.761956 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:35.761795 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/376b917f-91f0-4239-a1b3-eac12620d76e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/376b917f-91f0-4239-a1b3-eac12620d76e-istiod-ca-cert\") pod \"376b917f-91f0-4239-a1b3-eac12620d76e\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " Oct 30 13:40:35.761956 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:35.761844 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/376b917f-91f0-4239-a1b3-eac12620d76e-kube-api-access-4v82l\" (UniqueName: \"kubernetes.io/projected/376b917f-91f0-4239-a1b3-eac12620d76e-kube-api-access-4v82l\") pod \"376b917f-91f0-4239-a1b3-eac12620d76e\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " Oct 30 13:40:35.761956 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:35.761882 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/376b917f-91f0-4239-a1b3-eac12620d76e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/376b917f-91f0-4239-a1b3-eac12620d76e-istio-envoy\") pod \"376b917f-91f0-4239-a1b3-eac12620d76e\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " Oct 30 13:40:35.761956 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:35.761918 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/376b917f-91f0-4239-a1b3-eac12620d76e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/376b917f-91f0-4239-a1b3-eac12620d76e-workload-certs\") pod \"376b917f-91f0-4239-a1b3-eac12620d76e\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " Oct 30 13:40:35.762187 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:35.761995 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/376b917f-91f0-4239-a1b3-eac12620d76e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/376b917f-91f0-4239-a1b3-eac12620d76e-workload-socket\") pod \"376b917f-91f0-4239-a1b3-eac12620d76e\" (UID: \"376b917f-91f0-4239-a1b3-eac12620d76e\") " Oct 30 13:40:35.762187 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:35.762086 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/376b917f-91f0-4239-a1b3-eac12620d76e-credential-socket" pod "376b917f-91f0-4239-a1b3-eac12620d76e" (UID: "376b917f-91f0-4239-a1b3-eac12620d76e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:35.762383 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:35.762363 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/376b917f-91f0-4239-a1b3-eac12620d76e-istiod-ca-cert" pod "376b917f-91f0-4239-a1b3-eac12620d76e" (UID: "376b917f-91f0-4239-a1b3-eac12620d76e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:40:35.762452 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:35.762370 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/376b917f-91f0-4239-a1b3-eac12620d76e-workload-certs" pod "376b917f-91f0-4239-a1b3-eac12620d76e" (UID: "376b917f-91f0-4239-a1b3-eac12620d76e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:35.762452 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:35.762307 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/376b917f-91f0-4239-a1b3-eac12620d76e-istio-data" pod "376b917f-91f0-4239-a1b3-eac12620d76e" (UID: "376b917f-91f0-4239-a1b3-eac12620d76e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:35.762730 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:35.762702 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/376b917f-91f0-4239-a1b3-eac12620d76e-workload-socket" pod "376b917f-91f0-4239-a1b3-eac12620d76e" (UID: "376b917f-91f0-4239-a1b3-eac12620d76e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:35.762855 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:35.762793 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/376b917f-91f0-4239-a1b3-eac12620d76e-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:35.762855 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:35.762815 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/376b917f-91f0-4239-a1b3-eac12620d76e-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:35.762855 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:35.762830 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/376b917f-91f0-4239-a1b3-eac12620d76e-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:35.762855 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:35.762844 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/376b917f-91f0-4239-a1b3-eac12620d76e-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:35.764251 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:35.764212 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/376b917f-91f0-4239-a1b3-eac12620d76e-istio-podinfo" pod "376b917f-91f0-4239-a1b3-eac12620d76e" (UID: "376b917f-91f0-4239-a1b3-eac12620d76e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:40:35.764372 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:35.764294 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/376b917f-91f0-4239-a1b3-eac12620d76e-kube-api-access-4v82l" pod "376b917f-91f0-4239-a1b3-eac12620d76e" (UID: "376b917f-91f0-4239-a1b3-eac12620d76e"). InnerVolumeSpecName "kube-api-access-4v82l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:35.764461 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:35.764444 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/376b917f-91f0-4239-a1b3-eac12620d76e-istio-token" pod "376b917f-91f0-4239-a1b3-eac12620d76e" (UID: "376b917f-91f0-4239-a1b3-eac12620d76e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:35.764530 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:35.764471 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/376b917f-91f0-4239-a1b3-eac12620d76e-istio-envoy" pod "376b917f-91f0-4239-a1b3-eac12620d76e" (UID: "376b917f-91f0-4239-a1b3-eac12620d76e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:35.835464 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:35.835436 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8"] Oct 30 13:40:35.856461 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:35.856427 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k"] Oct 30 13:40:35.863784 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:35.863753 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/376b917f-91f0-4239-a1b3-eac12620d76e-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:35.863784 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:35.863784 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v82l\" (UniqueName: \"kubernetes.io/projected/376b917f-91f0-4239-a1b3-eac12620d76e-kube-api-access-4v82l\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:35.864011 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:35.863794 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/376b917f-91f0-4239-a1b3-eac12620d76e-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:35.864011 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:35.863803 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/376b917f-91f0-4239-a1b3-eac12620d76e-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:35.864011 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:35.863811 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/376b917f-91f0-4239-a1b3-eac12620d76e-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:35.864862 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:35.864833 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj"] Oct 30 13:40:36.008722 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:36.008688 4931 generic.go:356] "Generic (PLEG): container finished" podID="376b917f-91f0-4239-a1b3-eac12620d76e" containerID="03d514035c1e421392564bd70f0dcbed3806b16e4f6cf47d648d4c04a3b1df06" exitCode=0 Oct 30 13:40:36.008882 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:36.008745 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" Oct 30 13:40:36.008882 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:36.008761 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" event={"ID":"376b917f-91f0-4239-a1b3-eac12620d76e","Type":"ContainerDied","Data":"03d514035c1e421392564bd70f0dcbed3806b16e4f6cf47d648d4c04a3b1df06"} Oct 30 13:40:36.008882 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:36.008805 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd" event={"ID":"376b917f-91f0-4239-a1b3-eac12620d76e","Type":"ContainerDied","Data":"1219e3548d46706eda9989fada3812e5611be2208dabe080a7ffab1a5a9a3fe1"} Oct 30 13:40:36.008882 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:36.008827 4931 scope.go:122] "RemoveContainer" containerID="03d514035c1e421392564bd70f0dcbed3806b16e4f6cf47d648d4c04a3b1df06" Oct 30 13:40:36.023174 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:36.021658 4931 scope.go:122] "RemoveContainer" containerID="f8058e44f1c9d82de9e43e6588266533dc232c20d660fc6cc6cf1dcd4178031c" Oct 30 13:40:36.032059 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:36.032032 4931 scope.go:122] "RemoveContainer" containerID="03d514035c1e421392564bd70f0dcbed3806b16e4f6cf47d648d4c04a3b1df06" Oct 30 13:40:36.032347 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:40:36.032327 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03d514035c1e421392564bd70f0dcbed3806b16e4f6cf47d648d4c04a3b1df06\": container with ID starting with 03d514035c1e421392564bd70f0dcbed3806b16e4f6cf47d648d4c04a3b1df06 not found: ID does not exist" containerID="03d514035c1e421392564bd70f0dcbed3806b16e4f6cf47d648d4c04a3b1df06" Oct 30 13:40:36.032410 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:36.032356 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03d514035c1e421392564bd70f0dcbed3806b16e4f6cf47d648d4c04a3b1df06"} err="failed to get container status \"03d514035c1e421392564bd70f0dcbed3806b16e4f6cf47d648d4c04a3b1df06\": rpc error: code = NotFound desc = could not find container \"03d514035c1e421392564bd70f0dcbed3806b16e4f6cf47d648d4c04a3b1df06\": container with ID starting with 03d514035c1e421392564bd70f0dcbed3806b16e4f6cf47d648d4c04a3b1df06 not found: ID does not exist" Oct 30 13:40:36.032410 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:36.032374 4931 scope.go:122] "RemoveContainer" containerID="f8058e44f1c9d82de9e43e6588266533dc232c20d660fc6cc6cf1dcd4178031c" Oct 30 13:40:36.032608 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:40:36.032593 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8058e44f1c9d82de9e43e6588266533dc232c20d660fc6cc6cf1dcd4178031c\": container with ID starting with f8058e44f1c9d82de9e43e6588266533dc232c20d660fc6cc6cf1dcd4178031c not found: ID does not exist" containerID="f8058e44f1c9d82de9e43e6588266533dc232c20d660fc6cc6cf1dcd4178031c" Oct 30 13:40:36.032662 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:36.032612 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8058e44f1c9d82de9e43e6588266533dc232c20d660fc6cc6cf1dcd4178031c"} err="failed to get container status \"f8058e44f1c9d82de9e43e6588266533dc232c20d660fc6cc6cf1dcd4178031c\": rpc error: code = NotFound desc = could not find container \"f8058e44f1c9d82de9e43e6588266533dc232c20d660fc6cc6cf1dcd4178031c\": container with ID starting with f8058e44f1c9d82de9e43e6588266533dc232c20d660fc6cc6cf1dcd4178031c not found: ID does not exist" Oct 30 13:40:36.065650 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:36.065618 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd"] Oct 30 13:40:36.072672 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:36.072649 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd"] Oct 30 13:40:36.514795 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:36.514758 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv"] Oct 30 13:40:36.515001 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:36.514972 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" podUID="dc7b9784-abaf-439d-a30a-b19644f3c34a" containerName="istio-proxy" containerID="cri-o://619d13d3f7a48c75421811ec74f0998f0f738efcc57c27da3af1ecdb21826952" gracePeriod=30 Oct 30 13:40:37.290645 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:37.290604 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7a3603-4tpzs"] Oct 30 13:40:37.291312 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:37.291291 4931 cpu_manager.go:414] "containerMap: removing container" podUID="376b917f-91f0-4239-a1b3-eac12620d76e" containerName="istio-proxy" Oct 30 13:40:37.291422 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:37.291312 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="376b917f-91f0-4239-a1b3-eac12620d76e" containerName="istio-proxy" Oct 30 13:40:37.291422 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:37.291334 4931 cpu_manager.go:414] "containerMap: removing container" podUID="376b917f-91f0-4239-a1b3-eac12620d76e" containerName="istio-proxy" Oct 30 13:40:37.291422 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:37.291342 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="376b917f-91f0-4239-a1b3-eac12620d76e" containerName="istio-proxy" Oct 30 13:40:37.291422 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:37.291355 4931 cpu_manager.go:414] "containerMap: removing container" podUID="a00926d5-b69b-4a8a-b076-68ad63208f30" containerName="revoke-keys" Oct 30 13:40:37.291422 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:37.291363 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a00926d5-b69b-4a8a-b076-68ad63208f30" containerName="revoke-keys" Oct 30 13:40:37.291694 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:37.291494 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="376b917f-91f0-4239-a1b3-eac12620d76e" containerName="istio-proxy" Oct 30 13:40:37.291694 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:37.291511 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="376b917f-91f0-4239-a1b3-eac12620d76e" containerName="istio-proxy" Oct 30 13:40:37.291694 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:37.291521 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="a00926d5-b69b-4a8a-b076-68ad63208f30" containerName="revoke-keys" Oct 30 13:40:37.296469 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:37.296447 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7a3603-4tpzs" Oct 30 13:40:37.322007 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:37.321976 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7a3603-4tpzs"] Oct 30 13:40:37.362773 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:40:37.362735 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:40:37.364611 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:37.364528 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:40:37.364611 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:37.364575 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:40:37.386034 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:37.386000 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5db07c46-5a42-4453-a60d-d3cd9be08b32-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-7a3603-4tpzs\" (UID: \"5db07c46-5a42-4453-a60d-d3cd9be08b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7a3603-4tpzs" Oct 30 13:40:37.386222 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:37.386043 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qk2h\" (UniqueName: \"kubernetes.io/projected/5db07c46-5a42-4453-a60d-d3cd9be08b32-kube-api-access-2qk2h\") pod \"maas-api-revoke-keys-e2e-worker-w5-7a3603-4tpzs\" (UID: \"5db07c46-5a42-4453-a60d-d3cd9be08b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7a3603-4tpzs" Oct 30 13:40:37.386307 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:37.386213 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5db07c46-5a42-4453-a60d-d3cd9be08b32-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-7a3603-4tpzs\" (UID: \"5db07c46-5a42-4453-a60d-d3cd9be08b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7a3603-4tpzs" Oct 30 13:40:37.487427 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:37.487380 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qk2h\" (UniqueName: \"kubernetes.io/projected/5db07c46-5a42-4453-a60d-d3cd9be08b32-kube-api-access-2qk2h\") pod \"maas-api-revoke-keys-e2e-worker-w5-7a3603-4tpzs\" (UID: \"5db07c46-5a42-4453-a60d-d3cd9be08b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7a3603-4tpzs" Oct 30 13:40:37.487633 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:37.487560 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5db07c46-5a42-4453-a60d-d3cd9be08b32-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-7a3603-4tpzs\" (UID: \"5db07c46-5a42-4453-a60d-d3cd9be08b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7a3603-4tpzs" Oct 30 13:40:37.487700 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:37.487666 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5db07c46-5a42-4453-a60d-d3cd9be08b32-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-7a3603-4tpzs\" (UID: \"5db07c46-5a42-4453-a60d-d3cd9be08b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7a3603-4tpzs" Oct 30 13:40:37.488542 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:37.488510 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5db07c46-5a42-4453-a60d-d3cd9be08b32-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-7a3603-4tpzs\" (UID: \"5db07c46-5a42-4453-a60d-d3cd9be08b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7a3603-4tpzs" Oct 30 13:40:37.491856 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:37.491820 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5db07c46-5a42-4453-a60d-d3cd9be08b32-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-7a3603-4tpzs\" (UID: \"5db07c46-5a42-4453-a60d-d3cd9be08b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7a3603-4tpzs" Oct 30 13:40:37.514639 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:37.514613 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qk2h\" (UniqueName: \"kubernetes.io/projected/5db07c46-5a42-4453-a60d-d3cd9be08b32-kube-api-access-2qk2h\") pod \"maas-api-revoke-keys-e2e-worker-w5-7a3603-4tpzs\" (UID: \"5db07c46-5a42-4453-a60d-d3cd9be08b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7a3603-4tpzs" Oct 30 13:40:37.517213 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:37.517181 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="376b917f-91f0-4239-a1b3-eac12620d76e" path="/var/lib/kubelet/pods/376b917f-91f0-4239-a1b3-eac12620d76e/volumes" Oct 30 13:40:37.610597 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:37.610506 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7a3603-4tpzs" Oct 30 13:40:37.971416 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:37.971382 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7a3603-4tpzs"] Oct 30 13:40:38.028876 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:38.028849 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7a3603-4tpzs" event={"ID":"5db07c46-5a42-4453-a60d-d3cd9be08b32","Type":"ContainerStarted","Data":"de2d0eea0768ef54986f06d3c287570ac4b8c9f1d08bd9e6ae2d40d646ba4580"} Oct 30 13:40:38.913465 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:38.913426 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c90b9e-hr6ws"] Oct 30 13:40:38.926700 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:38.926665 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c90b9e-hr6ws"] Oct 30 13:40:39.037897 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:39.037851 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7a3603-4tpzs" event={"ID":"5db07c46-5a42-4453-a60d-d3cd9be08b32","Type":"ContainerStarted","Data":"3d7b7e97642f0106b1e2529dab5b1bc79466f0ea47b9a068cbf391ac9117dc47"} Oct 30 13:40:39.084692 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:39.084117 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7a3603-4tpzs" podStartSLOduration=2.084102559 podStartE2EDuration="2.084102559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:39.082579324 +0000 UTC m=+3337.686077770" watchObservedRunningTime="2026-10-30 13:40:39.084102559 +0000 UTC m=+3337.687601005" Oct 30 13:40:39.513379 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:39.513342 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06a57f38-0527-4855-b62d-2e18d16bf760" path="/var/lib/kubelet/pods/06a57f38-0527-4855-b62d-2e18d16bf760/volumes" Oct 30 13:40:40.047003 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:40.046966 4931 generic.go:356] "Generic (PLEG): container finished" podID="5db07c46-5a42-4453-a60d-d3cd9be08b32" containerID="3d7b7e97642f0106b1e2529dab5b1bc79466f0ea47b9a068cbf391ac9117dc47" exitCode=0 Oct 30 13:40:40.047448 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:40.047043 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7a3603-4tpzs" event={"ID":"5db07c46-5a42-4453-a60d-d3cd9be08b32","Type":"ContainerDied","Data":"3d7b7e97642f0106b1e2529dab5b1bc79466f0ea47b9a068cbf391ac9117dc47"} Oct 30 13:40:41.202384 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.202356 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7a3603-4tpzs" Oct 30 13:40:41.251383 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.251356 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5db07c46-5a42-4453-a60d-d3cd9be08b32-kube-api-access-2qk2h\" (UniqueName: \"kubernetes.io/projected/5db07c46-5a42-4453-a60d-d3cd9be08b32-kube-api-access-2qk2h\") pod \"5db07c46-5a42-4453-a60d-d3cd9be08b32\" (UID: \"5db07c46-5a42-4453-a60d-d3cd9be08b32\") " Oct 30 13:40:41.251564 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.251511 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5db07c46-5a42-4453-a60d-d3cd9be08b32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5db07c46-5a42-4453-a60d-d3cd9be08b32-maas-api-openshift-ca\") pod \"5db07c46-5a42-4453-a60d-d3cd9be08b32\" (UID: \"5db07c46-5a42-4453-a60d-d3cd9be08b32\") " Oct 30 13:40:41.251564 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.251552 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5db07c46-5a42-4453-a60d-d3cd9be08b32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5db07c46-5a42-4453-a60d-d3cd9be08b32-maas-api-xks-ca\") pod \"5db07c46-5a42-4453-a60d-d3cd9be08b32\" (UID: \"5db07c46-5a42-4453-a60d-d3cd9be08b32\") " Oct 30 13:40:41.251849 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.251827 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5db07c46-5a42-4453-a60d-d3cd9be08b32-maas-api-openshift-ca" pod "5db07c46-5a42-4453-a60d-d3cd9be08b32" (UID: "5db07c46-5a42-4453-a60d-d3cd9be08b32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:40:41.253505 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.253477 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db07c46-5a42-4453-a60d-d3cd9be08b32-kube-api-access-2qk2h" pod "5db07c46-5a42-4453-a60d-d3cd9be08b32" (UID: "5db07c46-5a42-4453-a60d-d3cd9be08b32"). InnerVolumeSpecName "kube-api-access-2qk2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:41.253642 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.253599 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5db07c46-5a42-4453-a60d-d3cd9be08b32-maas-api-xks-ca" pod "5db07c46-5a42-4453-a60d-d3cd9be08b32" (UID: "5db07c46-5a42-4453-a60d-d3cd9be08b32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:40:41.353130 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.353054 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5db07c46-5a42-4453-a60d-d3cd9be08b32-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:41.353130 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.353085 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5db07c46-5a42-4453-a60d-d3cd9be08b32-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:41.353130 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.353095 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qk2h\" (UniqueName: \"kubernetes.io/projected/5db07c46-5a42-4453-a60d-d3cd9be08b32-kube-api-access-2qk2h\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:41.546452 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.546424 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb"] Oct 30 13:40:41.802326 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.802291 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" Oct 30 13:40:41.858808 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.858775 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc7b9784-abaf-439d-a30a-b19644f3c34a-istio-token\" (UniqueName: \"kubernetes.io/projected/dc7b9784-abaf-439d-a30a-b19644f3c34a-istio-token\") pod \"dc7b9784-abaf-439d-a30a-b19644f3c34a\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " Oct 30 13:40:41.859004 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.858833 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc7b9784-abaf-439d-a30a-b19644f3c34a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc7b9784-abaf-439d-a30a-b19644f3c34a-istio-envoy\") pod \"dc7b9784-abaf-439d-a30a-b19644f3c34a\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " Oct 30 13:40:41.859004 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.858856 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc7b9784-abaf-439d-a30a-b19644f3c34a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc7b9784-abaf-439d-a30a-b19644f3c34a-workload-certs\") pod \"dc7b9784-abaf-439d-a30a-b19644f3c34a\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " Oct 30 13:40:41.859004 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.858885 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc7b9784-abaf-439d-a30a-b19644f3c34a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7b9784-abaf-439d-a30a-b19644f3c34a-workload-socket\") pod \"dc7b9784-abaf-439d-a30a-b19644f3c34a\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " Oct 30 13:40:41.859004 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.858957 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc7b9784-abaf-439d-a30a-b19644f3c34a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc7b9784-abaf-439d-a30a-b19644f3c34a-istiod-ca-cert\") pod \"dc7b9784-abaf-439d-a30a-b19644f3c34a\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " Oct 30 13:40:41.859004 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.858995 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc7b9784-abaf-439d-a30a-b19644f3c34a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc7b9784-abaf-439d-a30a-b19644f3c34a-istio-data\") pod \"dc7b9784-abaf-439d-a30a-b19644f3c34a\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " Oct 30 13:40:41.859342 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.859047 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc7b9784-abaf-439d-a30a-b19644f3c34a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7b9784-abaf-439d-a30a-b19644f3c34a-credential-socket\") pod \"dc7b9784-abaf-439d-a30a-b19644f3c34a\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " Oct 30 13:40:41.859342 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.859144 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc7b9784-abaf-439d-a30a-b19644f3c34a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc7b9784-abaf-439d-a30a-b19644f3c34a-istio-podinfo\") pod \"dc7b9784-abaf-439d-a30a-b19644f3c34a\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " Oct 30 13:40:41.859342 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.859201 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc7b9784-abaf-439d-a30a-b19644f3c34a-kube-api-access-k9mbl\" (UniqueName: \"kubernetes.io/projected/dc7b9784-abaf-439d-a30a-b19644f3c34a-kube-api-access-k9mbl\") pod \"dc7b9784-abaf-439d-a30a-b19644f3c34a\" (UID: \"dc7b9784-abaf-439d-a30a-b19644f3c34a\") " Oct 30 13:40:41.859342 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.859226 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc7b9784-abaf-439d-a30a-b19644f3c34a-workload-socket" pod "dc7b9784-abaf-439d-a30a-b19644f3c34a" (UID: "dc7b9784-abaf-439d-a30a-b19644f3c34a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:41.859342 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.859238 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc7b9784-abaf-439d-a30a-b19644f3c34a-workload-certs" pod "dc7b9784-abaf-439d-a30a-b19644f3c34a" (UID: "dc7b9784-abaf-439d-a30a-b19644f3c34a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:41.859606 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.859437 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc7b9784-abaf-439d-a30a-b19644f3c34a-istio-data" pod "dc7b9784-abaf-439d-a30a-b19644f3c34a" (UID: "dc7b9784-abaf-439d-a30a-b19644f3c34a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:41.859606 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.859501 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc7b9784-abaf-439d-a30a-b19644f3c34a-credential-socket" pod "dc7b9784-abaf-439d-a30a-b19644f3c34a" (UID: "dc7b9784-abaf-439d-a30a-b19644f3c34a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:41.859606 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.859535 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc7b9784-abaf-439d-a30a-b19644f3c34a-istiod-ca-cert" pod "dc7b9784-abaf-439d-a30a-b19644f3c34a" (UID: "dc7b9784-abaf-439d-a30a-b19644f3c34a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:40:41.859773 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.859752 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc7b9784-abaf-439d-a30a-b19644f3c34a-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:41.859837 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.859774 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7b9784-abaf-439d-a30a-b19644f3c34a-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:41.859837 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.859789 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc7b9784-abaf-439d-a30a-b19644f3c34a-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:41.859837 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.859802 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc7b9784-abaf-439d-a30a-b19644f3c34a-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:41.859837 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.859814 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7b9784-abaf-439d-a30a-b19644f3c34a-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:41.861364 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.861334 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc7b9784-abaf-439d-a30a-b19644f3c34a-istio-podinfo" pod "dc7b9784-abaf-439d-a30a-b19644f3c34a" (UID: "dc7b9784-abaf-439d-a30a-b19644f3c34a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:40:41.861545 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.861525 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc7b9784-abaf-439d-a30a-b19644f3c34a-istio-token" pod "dc7b9784-abaf-439d-a30a-b19644f3c34a" (UID: "dc7b9784-abaf-439d-a30a-b19644f3c34a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:41.861690 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.861673 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc7b9784-abaf-439d-a30a-b19644f3c34a-kube-api-access-k9mbl" pod "dc7b9784-abaf-439d-a30a-b19644f3c34a" (UID: "dc7b9784-abaf-439d-a30a-b19644f3c34a"). InnerVolumeSpecName "kube-api-access-k9mbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:41.863222 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.863196 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc7b9784-abaf-439d-a30a-b19644f3c34a-istio-envoy" pod "dc7b9784-abaf-439d-a30a-b19644f3c34a" (UID: "dc7b9784-abaf-439d-a30a-b19644f3c34a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:41.961164 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.961127 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc7b9784-abaf-439d-a30a-b19644f3c34a-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:41.961164 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.961159 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9mbl\" (UniqueName: \"kubernetes.io/projected/dc7b9784-abaf-439d-a30a-b19644f3c34a-kube-api-access-k9mbl\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:41.961164 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.961170 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc7b9784-abaf-439d-a30a-b19644f3c34a-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:41.961415 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:41.961180 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc7b9784-abaf-439d-a30a-b19644f3c34a-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:42.062125 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:42.062086 4931 generic.go:356] "Generic (PLEG): container finished" podID="dc7b9784-abaf-439d-a30a-b19644f3c34a" containerID="619d13d3f7a48c75421811ec74f0998f0f738efcc57c27da3af1ecdb21826952" exitCode=0 Oct 30 13:40:42.062363 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:42.062139 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" Oct 30 13:40:42.062363 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:42.062165 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" event={"ID":"dc7b9784-abaf-439d-a30a-b19644f3c34a","Type":"ContainerDied","Data":"619d13d3f7a48c75421811ec74f0998f0f738efcc57c27da3af1ecdb21826952"} Oct 30 13:40:42.062363 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:42.062218 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv" event={"ID":"dc7b9784-abaf-439d-a30a-b19644f3c34a","Type":"ContainerDied","Data":"a749055da6ea057296d9cc35aa5686acdf6649d5d69fab910967c93c50b49906"} Oct 30 13:40:42.062363 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:42.062244 4931 scope.go:122] "RemoveContainer" containerID="619d13d3f7a48c75421811ec74f0998f0f738efcc57c27da3af1ecdb21826952" Oct 30 13:40:42.064074 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:42.064045 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7a3603-4tpzs" event={"ID":"5db07c46-5a42-4453-a60d-d3cd9be08b32","Type":"ContainerDied","Data":"de2d0eea0768ef54986f06d3c287570ac4b8c9f1d08bd9e6ae2d40d646ba4580"} Oct 30 13:40:42.064210 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:42.064111 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7a3603-4tpzs" Oct 30 13:40:42.064210 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:42.064122 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de2d0eea0768ef54986f06d3c287570ac4b8c9f1d08bd9e6ae2d40d646ba4580" Oct 30 13:40:42.073238 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:42.073208 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k"] Oct 30 13:40:42.079815 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:42.079790 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz"] Oct 30 13:40:42.086406 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:42.086381 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj"] Oct 30 13:40:42.090238 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:42.090212 4931 scope.go:122] "RemoveContainer" containerID="619d13d3f7a48c75421811ec74f0998f0f738efcc57c27da3af1ecdb21826952" Oct 30 13:40:42.090578 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:40:42.090548 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"619d13d3f7a48c75421811ec74f0998f0f738efcc57c27da3af1ecdb21826952\": container with ID starting with 619d13d3f7a48c75421811ec74f0998f0f738efcc57c27da3af1ecdb21826952 not found: ID does not exist" containerID="619d13d3f7a48c75421811ec74f0998f0f738efcc57c27da3af1ecdb21826952" Oct 30 13:40:42.090703 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:42.090585 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"619d13d3f7a48c75421811ec74f0998f0f738efcc57c27da3af1ecdb21826952"} err="failed to get container status \"619d13d3f7a48c75421811ec74f0998f0f738efcc57c27da3af1ecdb21826952\": rpc error: code = NotFound desc = could not find container \"619d13d3f7a48c75421811ec74f0998f0f738efcc57c27da3af1ecdb21826952\": container with ID starting with 619d13d3f7a48c75421811ec74f0998f0f738efcc57c27da3af1ecdb21826952 not found: ID does not exist" Oct 30 13:40:42.266419 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:42.266318 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv"] Oct 30 13:40:42.271212 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:42.271109 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv"] Oct 30 13:40:43.119359 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:43.119327 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44e71584-xlwx7"] Oct 30 13:40:43.119815 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:43.119802 4931 cpu_manager.go:414] "containerMap: removing container" podUID="dc7b9784-abaf-439d-a30a-b19644f3c34a" containerName="istio-proxy" Oct 30 13:40:43.119856 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:43.119816 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc7b9784-abaf-439d-a30a-b19644f3c34a" containerName="istio-proxy" Oct 30 13:40:43.119856 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:43.119827 4931 cpu_manager.go:414] "containerMap: removing container" podUID="5db07c46-5a42-4453-a60d-d3cd9be08b32" containerName="revoke-keys" Oct 30 13:40:43.119856 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:43.119835 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5db07c46-5a42-4453-a60d-d3cd9be08b32" containerName="revoke-keys" Oct 30 13:40:43.119957 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:43.119919 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc7b9784-abaf-439d-a30a-b19644f3c34a" containerName="istio-proxy" Oct 30 13:40:43.119957 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:43.119928 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="5db07c46-5a42-4453-a60d-d3cd9be08b32" containerName="revoke-keys" Oct 30 13:40:43.124470 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:43.124446 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e71584-xlwx7" Oct 30 13:40:43.146212 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:43.146179 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44e71584-xlwx7"] Oct 30 13:40:43.155436 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:43.155412 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:40:43.155569 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:43.155438 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:40:43.176310 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:43.173596 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfwxp\" (UniqueName: \"kubernetes.io/projected/f9e56860-9840-4bd5-bf11-104f28578afe-kube-api-access-wfwxp\") pod \"maas-api-revoke-sub-44e71584-xlwx7\" (UID: \"f9e56860-9840-4bd5-bf11-104f28578afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e71584-xlwx7" Oct 30 13:40:43.176310 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:43.173818 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9e56860-9840-4bd5-bf11-104f28578afe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44e71584-xlwx7\" (UID: \"f9e56860-9840-4bd5-bf11-104f28578afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e71584-xlwx7" Oct 30 13:40:43.176310 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:43.173885 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9e56860-9840-4bd5-bf11-104f28578afe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44e71584-xlwx7\" (UID: \"f9e56860-9840-4bd5-bf11-104f28578afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e71584-xlwx7" Oct 30 13:40:43.275289 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:43.275235 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9e56860-9840-4bd5-bf11-104f28578afe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44e71584-xlwx7\" (UID: \"f9e56860-9840-4bd5-bf11-104f28578afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e71584-xlwx7" Oct 30 13:40:43.275768 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:43.275305 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfwxp\" (UniqueName: \"kubernetes.io/projected/f9e56860-9840-4bd5-bf11-104f28578afe-kube-api-access-wfwxp\") pod \"maas-api-revoke-sub-44e71584-xlwx7\" (UID: \"f9e56860-9840-4bd5-bf11-104f28578afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e71584-xlwx7" Oct 30 13:40:43.275768 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:43.275404 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9e56860-9840-4bd5-bf11-104f28578afe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44e71584-xlwx7\" (UID: \"f9e56860-9840-4bd5-bf11-104f28578afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e71584-xlwx7" Oct 30 13:40:43.276008 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:43.275987 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9e56860-9840-4bd5-bf11-104f28578afe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44e71584-xlwx7\" (UID: \"f9e56860-9840-4bd5-bf11-104f28578afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e71584-xlwx7" Oct 30 13:40:43.277606 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:43.277587 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9e56860-9840-4bd5-bf11-104f28578afe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44e71584-xlwx7\" (UID: \"f9e56860-9840-4bd5-bf11-104f28578afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e71584-xlwx7" Oct 30 13:40:43.301071 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:43.301037 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfwxp\" (UniqueName: \"kubernetes.io/projected/f9e56860-9840-4bd5-bf11-104f28578afe-kube-api-access-wfwxp\") pod \"maas-api-revoke-sub-44e71584-xlwx7\" (UID: \"f9e56860-9840-4bd5-bf11-104f28578afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e71584-xlwx7" Oct 30 13:40:43.438329 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:43.438237 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e71584-xlwx7" Oct 30 13:40:43.518909 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:43.518874 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc7b9784-abaf-439d-a30a-b19644f3c34a" path="/var/lib/kubelet/pods/dc7b9784-abaf-439d-a30a-b19644f3c34a/volumes" Oct 30 13:40:43.765725 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:40:43.765581 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9e56860_9840_4bd5_bf11_104f28578afe.slice/crio-7637242a0db2b732f101a7ed97078bfdf14f6a65e4d68d62e429faedf150ea47 WatchSource:0}: Error finding container 7637242a0db2b732f101a7ed97078bfdf14f6a65e4d68d62e429faedf150ea47: Status 404 returned error can't find the container with id 7637242a0db2b732f101a7ed97078bfdf14f6a65e4d68d62e429faedf150ea47 Oct 30 13:40:43.767330 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:43.767298 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44e71584-xlwx7"] Oct 30 13:40:44.080327 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:44.080292 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e71584-xlwx7" event={"ID":"f9e56860-9840-4bd5-bf11-104f28578afe","Type":"ContainerStarted","Data":"12ba743ef8521cf4c4dc8e898b6c2ef1cc496711a17b8888fbfae65d81e729ef"} Oct 30 13:40:44.080494 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:44.080349 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e71584-xlwx7" event={"ID":"f9e56860-9840-4bd5-bf11-104f28578afe","Type":"ContainerStarted","Data":"7637242a0db2b732f101a7ed97078bfdf14f6a65e4d68d62e429faedf150ea47"} Oct 30 13:40:44.127258 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:44.127207 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e71584-xlwx7" podStartSLOduration=1.127193039 podStartE2EDuration="1.127193039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:44.125533003 +0000 UTC m=+3342.729031449" watchObservedRunningTime="2026-10-30 13:40:44.127193039 +0000 UTC m=+3342.730691502" Oct 30 13:40:45.090608 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:45.090572 4931 generic.go:356] "Generic (PLEG): container finished" podID="f9e56860-9840-4bd5-bf11-104f28578afe" containerID="12ba743ef8521cf4c4dc8e898b6c2ef1cc496711a17b8888fbfae65d81e729ef" exitCode=0 Oct 30 13:40:45.091055 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:45.090624 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e71584-xlwx7" event={"ID":"f9e56860-9840-4bd5-bf11-104f28578afe","Type":"ContainerDied","Data":"12ba743ef8521cf4c4dc8e898b6c2ef1cc496711a17b8888fbfae65d81e729ef"} Oct 30 13:40:46.249644 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:46.249603 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e71584-xlwx7" Oct 30 13:40:46.310106 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:46.310069 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9e56860-9840-4bd5-bf11-104f28578afe-kube-api-access-wfwxp\" (UniqueName: \"kubernetes.io/projected/f9e56860-9840-4bd5-bf11-104f28578afe-kube-api-access-wfwxp\") pod \"f9e56860-9840-4bd5-bf11-104f28578afe\" (UID: \"f9e56860-9840-4bd5-bf11-104f28578afe\") " Oct 30 13:40:46.310345 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:46.310154 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9e56860-9840-4bd5-bf11-104f28578afe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9e56860-9840-4bd5-bf11-104f28578afe-maas-api-xks-ca\") pod \"f9e56860-9840-4bd5-bf11-104f28578afe\" (UID: \"f9e56860-9840-4bd5-bf11-104f28578afe\") " Oct 30 13:40:46.310345 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:46.310191 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9e56860-9840-4bd5-bf11-104f28578afe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9e56860-9840-4bd5-bf11-104f28578afe-maas-api-openshift-ca\") pod \"f9e56860-9840-4bd5-bf11-104f28578afe\" (UID: \"f9e56860-9840-4bd5-bf11-104f28578afe\") " Oct 30 13:40:46.310683 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:46.310647 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9e56860-9840-4bd5-bf11-104f28578afe-maas-api-openshift-ca" pod "f9e56860-9840-4bd5-bf11-104f28578afe" (UID: "f9e56860-9840-4bd5-bf11-104f28578afe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:40:46.310811 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:46.310785 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9e56860-9840-4bd5-bf11-104f28578afe-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:46.312354 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:46.312323 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9e56860-9840-4bd5-bf11-104f28578afe-maas-api-xks-ca" pod "f9e56860-9840-4bd5-bf11-104f28578afe" (UID: "f9e56860-9840-4bd5-bf11-104f28578afe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:40:46.312455 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:46.312429 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9e56860-9840-4bd5-bf11-104f28578afe-kube-api-access-wfwxp" pod "f9e56860-9840-4bd5-bf11-104f28578afe" (UID: "f9e56860-9840-4bd5-bf11-104f28578afe"). InnerVolumeSpecName "kube-api-access-wfwxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:46.411819 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:46.411742 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfwxp\" (UniqueName: \"kubernetes.io/projected/f9e56860-9840-4bd5-bf11-104f28578afe-kube-api-access-wfwxp\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:46.411819 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:46.411772 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9e56860-9840-4bd5-bf11-104f28578afe-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:46.547581 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:46.547520 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb" podUID="440d993b-e62e-4284-955d-2e79adccec2e" containerName="payload-pre-processing" containerID="cri-o://f437a14b7e3a5be4eb14ad1f26a65ad40adbaa59dd34f6b1a5823245b7081ce2" gracePeriod=25 Oct 30 13:40:46.787721 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:46.787675 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:40:47.106240 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:47.106163 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e71584-xlwx7" Oct 30 13:40:47.106418 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:47.106166 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e71584-xlwx7" event={"ID":"f9e56860-9840-4bd5-bf11-104f28578afe","Type":"ContainerDied","Data":"7637242a0db2b732f101a7ed97078bfdf14f6a65e4d68d62e429faedf150ea47"} Oct 30 13:40:47.106418 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:47.106316 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7637242a0db2b732f101a7ed97078bfdf14f6a65e4d68d62e429faedf150ea47" Oct 30 13:40:47.914997 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:47.914957 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt"] Oct 30 13:40:48.507525 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:48.506837 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h"] Oct 30 13:40:48.507525 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:48.507072 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" podUID="fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9" containerName="istio-proxy" containerID="cri-o://f72beba7d76820188fe8ca1245ec15c65c499707d539fe2d36631615633b6fb0" gracePeriod=30 Oct 30 13:40:50.856954 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:50.856887 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" podUID="7ad1989a-c73f-4647-9e50-855605bdb08e" containerName="main" containerID="cri-o://0af1f8fcf29ba69dde70ce366df699a639fbe7aa39be5feb7aff4f8b3b3fb6d3" gracePeriod=45 Oct 30 13:40:50.876732 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:50.876692 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" podUID="e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" containerName="main" containerID="cri-o://b92efe13a80793dae4efc15df9ee1dfb730fa6366ef1f772b0fcd1b717656636" gracePeriod=45 Oct 30 13:40:50.884791 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:50.884758 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" podUID="606b252b-881c-4d40-8ba9-fc4ed937a995" containerName="main" containerID="cri-o://7d68128c0f65505a22952b820ae87b9c0472ab6b2a1a75b8cdac2878b6eae845" gracePeriod=45 Oct 30 13:40:51.034397 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.034333 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" podUID="7ad1989a-c73f-4647-9e50-855605bdb08e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": dial tcp 10.132.0.84:8000: connect: connection refused" Oct 30 13:40:51.148234 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.148198 4931 generic.go:356] "Generic (PLEG): container finished" podID="606b252b-881c-4d40-8ba9-fc4ed937a995" containerID="7d68128c0f65505a22952b820ae87b9c0472ab6b2a1a75b8cdac2878b6eae845" exitCode=0 Oct 30 13:40:51.148463 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.148309 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" event={"ID":"606b252b-881c-4d40-8ba9-fc4ed937a995","Type":"ContainerDied","Data":"7d68128c0f65505a22952b820ae87b9c0472ab6b2a1a75b8cdac2878b6eae845"} Oct 30 13:40:51.151455 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.151422 4931 generic.go:356] "Generic (PLEG): container finished" podID="7ad1989a-c73f-4647-9e50-855605bdb08e" containerID="0af1f8fcf29ba69dde70ce366df699a639fbe7aa39be5feb7aff4f8b3b3fb6d3" exitCode=0 Oct 30 13:40:51.151591 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.151548 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" event={"ID":"7ad1989a-c73f-4647-9e50-855605bdb08e","Type":"ContainerDied","Data":"0af1f8fcf29ba69dde70ce366df699a639fbe7aa39be5feb7aff4f8b3b3fb6d3"} Oct 30 13:40:51.153602 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.153574 4931 generic.go:356] "Generic (PLEG): container finished" podID="e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" containerID="b92efe13a80793dae4efc15df9ee1dfb730fa6366ef1f772b0fcd1b717656636" exitCode=0 Oct 30 13:40:51.153784 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.153768 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" event={"ID":"e52bb4f7-ceff-4e39-91ce-a8f48b205a0a","Type":"ContainerDied","Data":"b92efe13a80793dae4efc15df9ee1dfb730fa6366ef1f772b0fcd1b717656636"} Oct 30 13:40:51.171885 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.171862 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:40:51.195182 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.195153 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:40:51.198953 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.198933 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 13:40:51.272601 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.272567 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad1989a-c73f-4647-9e50-855605bdb08e-home\" (UniqueName: \"kubernetes.io/empty-dir/7ad1989a-c73f-4647-9e50-855605bdb08e-home\") pod \"7ad1989a-c73f-4647-9e50-855605bdb08e\" (UID: \"7ad1989a-c73f-4647-9e50-855605bdb08e\") " Oct 30 13:40:51.272820 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.272623 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-kube-api-access-kxhpb\" (UniqueName: \"kubernetes.io/projected/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-kube-api-access-kxhpb\") pod \"e52bb4f7-ceff-4e39-91ce-a8f48b205a0a\" (UID: \"e52bb4f7-ceff-4e39-91ce-a8f48b205a0a\") " Oct 30 13:40:51.272866 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.272842 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-model-cache\") pod \"e52bb4f7-ceff-4e39-91ce-a8f48b205a0a\" (UID: \"e52bb4f7-ceff-4e39-91ce-a8f48b205a0a\") " Oct 30 13:40:51.272911 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.272873 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-home\" (UniqueName: \"kubernetes.io/empty-dir/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-home\") pod \"e52bb4f7-ceff-4e39-91ce-a8f48b205a0a\" (UID: \"e52bb4f7-ceff-4e39-91ce-a8f48b205a0a\") " Oct 30 13:40:51.273132 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.273118 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/606b252b-881c-4d40-8ba9-fc4ed937a995-kube-api-access-2jjd2\" (UniqueName: \"kubernetes.io/projected/606b252b-881c-4d40-8ba9-fc4ed937a995-kube-api-access-2jjd2\") pod \"606b252b-881c-4d40-8ba9-fc4ed937a995\" (UID: \"606b252b-881c-4d40-8ba9-fc4ed937a995\") " Oct 30 13:40:51.273180 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.273145 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/606b252b-881c-4d40-8ba9-fc4ed937a995-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/606b252b-881c-4d40-8ba9-fc4ed937a995-tmp-dir\") pod \"606b252b-881c-4d40-8ba9-fc4ed937a995\" (UID: \"606b252b-881c-4d40-8ba9-fc4ed937a995\") " Oct 30 13:40:51.273400 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.273168 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ad1989a-c73f-4647-9e50-855605bdb08e-tls-certs\" (UniqueName: \"kubernetes.io/secret/7ad1989a-c73f-4647-9e50-855605bdb08e-tls-certs\") pod \"7ad1989a-c73f-4647-9e50-855605bdb08e\" (UID: \"7ad1989a-c73f-4647-9e50-855605bdb08e\") " Oct 30 13:40:51.273537 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.273414 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad1989a-c73f-4647-9e50-855605bdb08e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ad1989a-c73f-4647-9e50-855605bdb08e-model-cache\") pod \"7ad1989a-c73f-4647-9e50-855605bdb08e\" (UID: \"7ad1989a-c73f-4647-9e50-855605bdb08e\") " Oct 30 13:40:51.273537 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.273467 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/606b252b-881c-4d40-8ba9-fc4ed937a995-home\" (UniqueName: \"kubernetes.io/empty-dir/606b252b-881c-4d40-8ba9-fc4ed937a995-home\") pod \"606b252b-881c-4d40-8ba9-fc4ed937a995\" (UID: \"606b252b-881c-4d40-8ba9-fc4ed937a995\") " Oct 30 13:40:51.273743 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.273730 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad1989a-c73f-4647-9e50-855605bdb08e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ad1989a-c73f-4647-9e50-855605bdb08e-tmp-dir\") pod \"7ad1989a-c73f-4647-9e50-855605bdb08e\" (UID: \"7ad1989a-c73f-4647-9e50-855605bdb08e\") " Oct 30 13:40:51.273811 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.273755 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad1989a-c73f-4647-9e50-855605bdb08e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ad1989a-c73f-4647-9e50-855605bdb08e-dshm\") pod \"7ad1989a-c73f-4647-9e50-855605bdb08e\" (UID: \"7ad1989a-c73f-4647-9e50-855605bdb08e\") " Oct 30 13:40:51.274039 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.274022 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-dshm\") pod \"e52bb4f7-ceff-4e39-91ce-a8f48b205a0a\" (UID: \"e52bb4f7-ceff-4e39-91ce-a8f48b205a0a\") " Oct 30 13:40:51.274102 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.274073 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ad1989a-c73f-4647-9e50-855605bdb08e-kube-api-access-nhx74\" (UniqueName: \"kubernetes.io/projected/7ad1989a-c73f-4647-9e50-855605bdb08e-kube-api-access-nhx74\") pod \"7ad1989a-c73f-4647-9e50-855605bdb08e\" (UID: \"7ad1989a-c73f-4647-9e50-855605bdb08e\") " Oct 30 13:40:51.274165 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.274122 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-tmp-dir\") pod \"e52bb4f7-ceff-4e39-91ce-a8f48b205a0a\" (UID: \"e52bb4f7-ceff-4e39-91ce-a8f48b205a0a\") " Oct 30 13:40:51.274448 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.274428 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/606b252b-881c-4d40-8ba9-fc4ed937a995-tls-certs\" (UniqueName: \"kubernetes.io/secret/606b252b-881c-4d40-8ba9-fc4ed937a995-tls-certs\") pod \"606b252b-881c-4d40-8ba9-fc4ed937a995\" (UID: \"606b252b-881c-4d40-8ba9-fc4ed937a995\") " Oct 30 13:40:51.274574 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.274484 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/606b252b-881c-4d40-8ba9-fc4ed937a995-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/606b252b-881c-4d40-8ba9-fc4ed937a995-model-cache\") pod \"606b252b-881c-4d40-8ba9-fc4ed937a995\" (UID: \"606b252b-881c-4d40-8ba9-fc4ed937a995\") " Oct 30 13:40:51.274806 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.274776 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/606b252b-881c-4d40-8ba9-fc4ed937a995-dshm\" (UniqueName: \"kubernetes.io/empty-dir/606b252b-881c-4d40-8ba9-fc4ed937a995-dshm\") pod \"606b252b-881c-4d40-8ba9-fc4ed937a995\" (UID: \"606b252b-881c-4d40-8ba9-fc4ed937a995\") " Oct 30 13:40:51.274939 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.274920 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-tls-certs\" (UniqueName: \"kubernetes.io/secret/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-tls-certs\") pod \"e52bb4f7-ceff-4e39-91ce-a8f48b205a0a\" (UID: \"e52bb4f7-ceff-4e39-91ce-a8f48b205a0a\") " Oct 30 13:40:51.282208 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.280612 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad1989a-c73f-4647-9e50-855605bdb08e-model-cache" pod "7ad1989a-c73f-4647-9e50-855605bdb08e" (UID: "7ad1989a-c73f-4647-9e50-855605bdb08e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:51.282208 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.280798 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad1989a-c73f-4647-9e50-855605bdb08e-tmp-dir" pod "7ad1989a-c73f-4647-9e50-855605bdb08e" (UID: "7ad1989a-c73f-4647-9e50-855605bdb08e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:51.282208 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.280848 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-tmp-dir" pod "e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" (UID: "e52bb4f7-ceff-4e39-91ce-a8f48b205a0a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:51.282208 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.281214 4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ad1989a-c73f-4647-9e50-855605bdb08e-model-cache\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:51.282208 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.281247 4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ad1989a-c73f-4647-9e50-855605bdb08e-tmp-dir\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:51.282208 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.281285 4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-tmp-dir\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:51.282208 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.281290 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/606b252b-881c-4d40-8ba9-fc4ed937a995-home" pod "606b252b-881c-4d40-8ba9-fc4ed937a995" (UID: "606b252b-881c-4d40-8ba9-fc4ed937a995"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:51.282208 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.281612 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/606b252b-881c-4d40-8ba9-fc4ed937a995-model-cache" pod "606b252b-881c-4d40-8ba9-fc4ed937a995" (UID: "606b252b-881c-4d40-8ba9-fc4ed937a995"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:51.282208 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.281868 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-home" pod "e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" (UID: "e52bb4f7-ceff-4e39-91ce-a8f48b205a0a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:51.282208 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.282082 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad1989a-c73f-4647-9e50-855605bdb08e-home" pod "7ad1989a-c73f-4647-9e50-855605bdb08e" (UID: "7ad1989a-c73f-4647-9e50-855605bdb08e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:51.283654 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.283452 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/606b252b-881c-4d40-8ba9-fc4ed937a995-tmp-dir" pod "606b252b-881c-4d40-8ba9-fc4ed937a995" (UID: "606b252b-881c-4d40-8ba9-fc4ed937a995"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:51.284195 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.284166 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-dshm" pod "e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" (UID: "e52bb4f7-ceff-4e39-91ce-a8f48b205a0a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:51.285954 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.285928 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-model-cache" pod "e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" (UID: "e52bb4f7-ceff-4e39-91ce-a8f48b205a0a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:51.286143 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.286120 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/606b252b-881c-4d40-8ba9-fc4ed937a995-dshm" pod "606b252b-881c-4d40-8ba9-fc4ed937a995" (UID: "606b252b-881c-4d40-8ba9-fc4ed937a995"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:51.288340 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.286616 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/606b252b-881c-4d40-8ba9-fc4ed937a995-kube-api-access-2jjd2" pod "606b252b-881c-4d40-8ba9-fc4ed937a995" (UID: "606b252b-881c-4d40-8ba9-fc4ed937a995"). InnerVolumeSpecName "kube-api-access-2jjd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:51.288340 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.286705 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad1989a-c73f-4647-9e50-855605bdb08e-kube-api-access-nhx74" pod "7ad1989a-c73f-4647-9e50-855605bdb08e" (UID: "7ad1989a-c73f-4647-9e50-855605bdb08e"). InnerVolumeSpecName "kube-api-access-nhx74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:51.288340 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.286875 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/606b252b-881c-4d40-8ba9-fc4ed937a995-tls-certs" pod "606b252b-881c-4d40-8ba9-fc4ed937a995" (UID: "606b252b-881c-4d40-8ba9-fc4ed937a995"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:40:51.294377 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.293774 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-kube-api-access-kxhpb" pod "e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" (UID: "e52bb4f7-ceff-4e39-91ce-a8f48b205a0a"). InnerVolumeSpecName "kube-api-access-kxhpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:51.294377 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.293987 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-tls-certs" pod "e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" (UID: "e52bb4f7-ceff-4e39-91ce-a8f48b205a0a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:40:51.298956 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.298835 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad1989a-c73f-4647-9e50-855605bdb08e-dshm" pod "7ad1989a-c73f-4647-9e50-855605bdb08e" (UID: "7ad1989a-c73f-4647-9e50-855605bdb08e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:51.300665 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.300643 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ad1989a-c73f-4647-9e50-855605bdb08e-tls-certs" pod "7ad1989a-c73f-4647-9e50-855605bdb08e" (UID: "7ad1989a-c73f-4647-9e50-855605bdb08e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:40:51.382744 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.382698 4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/606b252b-881c-4d40-8ba9-fc4ed937a995-home\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:51.382744 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.382742 4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ad1989a-c73f-4647-9e50-855605bdb08e-dshm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:51.382744 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.382751 4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-dshm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:51.382985 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.382763 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhx74\" (UniqueName: \"kubernetes.io/projected/7ad1989a-c73f-4647-9e50-855605bdb08e-kube-api-access-nhx74\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:51.382985 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.382774 4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/606b252b-881c-4d40-8ba9-fc4ed937a995-tls-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:51.382985 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.382784 4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/606b252b-881c-4d40-8ba9-fc4ed937a995-model-cache\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:51.382985 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.382792 4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/606b252b-881c-4d40-8ba9-fc4ed937a995-dshm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:51.382985 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.382799 4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-tls-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:51.382985 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.382807 4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ad1989a-c73f-4647-9e50-855605bdb08e-home\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:51.382985 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.382814 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxhpb\" (UniqueName: \"kubernetes.io/projected/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-kube-api-access-kxhpb\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:51.382985 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.382823 4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-model-cache\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:51.382985 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.382831 4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a-home\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:51.382985 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.382839 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jjd2\" (UniqueName: \"kubernetes.io/projected/606b252b-881c-4d40-8ba9-fc4ed937a995-kube-api-access-2jjd2\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:51.382985 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.382847 4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/606b252b-881c-4d40-8ba9-fc4ed937a995-tmp-dir\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:51.382985 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:51.382855 4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ad1989a-c73f-4647-9e50-855605bdb08e-tls-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:52.164392 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:52.164304 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" Oct 30 13:40:52.164819 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:52.164290 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj" event={"ID":"606b252b-881c-4d40-8ba9-fc4ed937a995","Type":"ContainerDied","Data":"cf907cf68a3b69579ea545e3f069fc6c02d3c482084a60a115bf7363f51c0198"} Oct 30 13:40:52.164819 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:52.164565 4931 scope.go:122] "RemoveContainer" containerID="7d68128c0f65505a22952b820ae87b9c0472ab6b2a1a75b8cdac2878b6eae845" Oct 30 13:40:52.166116 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:52.166058 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" Oct 30 13:40:52.166116 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:52.166070 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8" event={"ID":"7ad1989a-c73f-4647-9e50-855605bdb08e","Type":"ContainerDied","Data":"0b4be3edfc5ed83ef40be03912f53313fab8c3385241223c30d8218e297c38b5"} Oct 30 13:40:52.167891 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:52.167867 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" Oct 30 13:40:52.168057 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:52.168025 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k" event={"ID":"e52bb4f7-ceff-4e39-91ce-a8f48b205a0a","Type":"ContainerDied","Data":"59600529f791ea7a12f67cd90131440f8cc7cdb40720f597e022966a1f9d1a22"} Oct 30 13:40:52.179325 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:52.179306 4931 scope.go:122] "RemoveContainer" containerID="0af1f8fcf29ba69dde70ce366df699a639fbe7aa39be5feb7aff4f8b3b3fb6d3" Oct 30 13:40:52.188388 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:52.188369 4931 scope.go:122] "RemoveContainer" containerID="b92efe13a80793dae4efc15df9ee1dfb730fa6366ef1f772b0fcd1b717656636" Oct 30 13:40:52.222010 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:52.221984 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj"] Oct 30 13:40:52.235933 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:52.235908 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-dd90cb/simulator-w6-dd90cb-kserve-77cd5c969-s8xqj"] Oct 30 13:40:52.272026 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:52.271998 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8"] Oct 30 13:40:52.278449 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:52.278394 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-dd90cb/distinct-2-w6-dd90cb-kserve-77499b8ff6-89cf8"] Oct 30 13:40:52.314787 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:52.314747 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k"] Oct 30 13:40:52.317479 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:52.317449 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-dd90cb/premium-w6-dd90cb-kserve-5fd959df56-7rz9k"] Oct 30 13:40:52.916040 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:52.915960 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt" podUID="03309fcc-c2dd-4424-9deb-c34596cd1558" containerName="payload-processing" containerID="cri-o://ff901eab67367b166da25cdddbf18beb67acbc1c54924922c824008bce2c248e" gracePeriod=25 Oct 30 13:40:53.177781 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.177720 4931 generic.go:356] "Generic (PLEG): container finished" podID="03309fcc-c2dd-4424-9deb-c34596cd1558" containerID="ff901eab67367b166da25cdddbf18beb67acbc1c54924922c824008bce2c248e" exitCode=0 Oct 30 13:40:53.178152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.177794 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt" event={"ID":"03309fcc-c2dd-4424-9deb-c34596cd1558","Type":"ContainerDied","Data":"ff901eab67367b166da25cdddbf18beb67acbc1c54924922c824008bce2c248e"} Oct 30 13:40:53.178152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.177860 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt" event={"ID":"03309fcc-c2dd-4424-9deb-c34596cd1558","Type":"ContainerDied","Data":"7c887ec035061014e2ccf3ddbeb41d6bc5d9d72dac92703c5f2a957ffeee2bd2"} Oct 30 13:40:53.178152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.177889 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c887ec035061014e2ccf3ddbeb41d6bc5d9d72dac92703c5f2a957ffeee2bd2" Oct 30 13:40:53.189325 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.185760 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt" Oct 30 13:40:53.306070 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.306039 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03309fcc-c2dd-4424-9deb-c34596cd1558-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03309fcc-c2dd-4424-9deb-c34596cd1558-plugins-config-volume\") pod \"03309fcc-c2dd-4424-9deb-c34596cd1558\" (UID: \"03309fcc-c2dd-4424-9deb-c34596cd1558\") " Oct 30 13:40:53.306248 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.306141 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03309fcc-c2dd-4424-9deb-c34596cd1558-kube-api-access-b2tbr\" (UniqueName: \"kubernetes.io/projected/03309fcc-c2dd-4424-9deb-c34596cd1558-kube-api-access-b2tbr\") pod \"03309fcc-c2dd-4424-9deb-c34596cd1558\" (UID: \"03309fcc-c2dd-4424-9deb-c34596cd1558\") " Oct 30 13:40:53.306450 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.306427 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03309fcc-c2dd-4424-9deb-c34596cd1558-plugins-config-volume" pod "03309fcc-c2dd-4424-9deb-c34596cd1558" (UID: "03309fcc-c2dd-4424-9deb-c34596cd1558"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:40:53.306526 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.306514 4931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03309fcc-c2dd-4424-9deb-c34596cd1558-plugins-config-volume\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:53.308289 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.308254 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03309fcc-c2dd-4424-9deb-c34596cd1558-kube-api-access-b2tbr" pod "03309fcc-c2dd-4424-9deb-c34596cd1558" (UID: "03309fcc-c2dd-4424-9deb-c34596cd1558"). InnerVolumeSpecName "kube-api-access-b2tbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:53.367397 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.367360 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb" podUID="440d993b-e62e-4284-955d-2e79adccec2e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:40:53.407280 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.407243 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2tbr\" (UniqueName: \"kubernetes.io/projected/03309fcc-c2dd-4424-9deb-c34596cd1558-kube-api-access-b2tbr\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:53.468072 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.468027 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8"] Oct 30 13:40:53.468488 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.468455 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" podUID="fc65620b-073b-4e13-82da-38526fb90da3" containerName="istio-proxy" containerID="cri-o://decf2dabff7f6ac00617d6984a38c747dafc1a911633f93c4f2b5cef67483bae" gracePeriod=30 Oct 30 13:40:53.512458 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.512432 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="606b252b-881c-4d40-8ba9-fc4ed937a995" path="/var/lib/kubelet/pods/606b252b-881c-4d40-8ba9-fc4ed937a995/volumes" Oct 30 13:40:53.512883 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.512868 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ad1989a-c73f-4647-9e50-855605bdb08e" path="/var/lib/kubelet/pods/7ad1989a-c73f-4647-9e50-855605bdb08e/volumes" Oct 30 13:40:53.513257 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.513244 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" path="/var/lib/kubelet/pods/e52bb4f7-ceff-4e39-91ce-a8f48b205a0a/volumes" Oct 30 13:40:53.759218 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.759193 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" Oct 30 13:40:53.811008 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.810974 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-workload-certs\") pod \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " Oct 30 13:40:53.811251 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.811037 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-istiod-ca-cert\") pod \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " Oct 30 13:40:53.811251 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.811097 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-workload-socket\") pod \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " Oct 30 13:40:53.811251 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.811129 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-credential-socket\") pod \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " Oct 30 13:40:53.811251 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.811162 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-istio-token\" (UniqueName: \"kubernetes.io/projected/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-istio-token\") pod \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " Oct 30 13:40:53.811251 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.811230 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-istio-data\") pod \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " Oct 30 13:40:53.811570 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.811317 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-workload-certs" pod "fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9" (UID: "fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:53.811570 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.811341 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-istio-podinfo\") pod \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " Oct 30 13:40:53.811570 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.811372 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-kube-api-access-9bw9s\" (UniqueName: \"kubernetes.io/projected/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-kube-api-access-9bw9s\") pod \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " Oct 30 13:40:53.811570 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.811415 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-istio-envoy\") pod \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\" (UID: \"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9\") " Oct 30 13:40:53.811570 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.811546 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-istiod-ca-cert" pod "fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9" (UID: "fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:40:53.811825 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.811806 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-credential-socket" pod "fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9" (UID: "fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:53.811922 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.811907 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:53.811955 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.811931 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:53.811955 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.811945 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:53.812049 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.811908 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-istio-data" pod "fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9" (UID: "fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:53.812049 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.811959 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-workload-socket" pod "fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9" (UID: "fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:53.819718 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.819688 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-istio-token" pod "fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9" (UID: "fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:53.819718 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.819710 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-kube-api-access-9bw9s" pod "fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9" (UID: "fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9"). InnerVolumeSpecName "kube-api-access-9bw9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:53.819860 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.819719 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-istio-podinfo" pod "fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9" (UID: "fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:40:53.819860 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.819820 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-istio-envoy" pod "fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9" (UID: "fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:53.912842 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.912803 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:53.912842 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.912837 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:53.913030 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.912852 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:53.913030 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.912863 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:53.913030 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.912875 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bw9s\" (UniqueName: \"kubernetes.io/projected/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-kube-api-access-9bw9s\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:53.913030 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:53.912887 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:54.196258 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:54.196219 4931 generic.go:356] "Generic (PLEG): container finished" podID="fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9" containerID="f72beba7d76820188fe8ca1245ec15c65c499707d539fe2d36631615633b6fb0" exitCode=0 Oct 30 13:40:54.196666 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:54.196306 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" event={"ID":"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9","Type":"ContainerDied","Data":"f72beba7d76820188fe8ca1245ec15c65c499707d539fe2d36631615633b6fb0"} Oct 30 13:40:54.196666 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:54.196327 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" Oct 30 13:40:54.196666 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:54.196387 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h" event={"ID":"fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9","Type":"ContainerDied","Data":"9db080bda909a79e21a4da49a194e9fd0e8a3782ef05e060e0f25733ddf6e046"} Oct 30 13:40:54.196666 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:54.196411 4931 scope.go:122] "RemoveContainer" containerID="f72beba7d76820188fe8ca1245ec15c65c499707d539fe2d36631615633b6fb0" Oct 30 13:40:54.196666 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:54.196524 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt" Oct 30 13:40:54.208460 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:54.208439 4931 scope.go:122] "RemoveContainer" containerID="f72beba7d76820188fe8ca1245ec15c65c499707d539fe2d36631615633b6fb0" Oct 30 13:40:54.208770 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:40:54.208747 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f72beba7d76820188fe8ca1245ec15c65c499707d539fe2d36631615633b6fb0\": container with ID starting with f72beba7d76820188fe8ca1245ec15c65c499707d539fe2d36631615633b6fb0 not found: ID does not exist" containerID="f72beba7d76820188fe8ca1245ec15c65c499707d539fe2d36631615633b6fb0" Oct 30 13:40:54.208873 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:54.208774 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f72beba7d76820188fe8ca1245ec15c65c499707d539fe2d36631615633b6fb0"} err="failed to get container status \"f72beba7d76820188fe8ca1245ec15c65c499707d539fe2d36631615633b6fb0\": rpc error: code = NotFound desc = could not find container \"f72beba7d76820188fe8ca1245ec15c65c499707d539fe2d36631615633b6fb0\": container with ID starting with f72beba7d76820188fe8ca1245ec15c65c499707d539fe2d36631615633b6fb0 not found: ID does not exist" Oct 30 13:40:54.272122 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:54.272091 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt"] Oct 30 13:40:54.279632 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:54.279606 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt"] Oct 30 13:40:54.323659 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:54.323624 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h"] Oct 30 13:40:54.326206 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:54.326186 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h"] Oct 30 13:40:55.512396 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:55.512363 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03309fcc-c2dd-4424-9deb-c34596cd1558" path="/var/lib/kubelet/pods/03309fcc-c2dd-4424-9deb-c34596cd1558/volumes" Oct 30 13:40:55.512857 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:55.512843 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9" path="/var/lib/kubelet/pods/fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9/volumes" Oct 30 13:40:57.103207 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.103167 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" podUID="f3b342db-9a01-46ad-b476-bfc912d9ce1c" containerName="main" containerID="cri-o://79e26cbf888e58bce0368a612e9795d8d1056f87ab4e441c8a042ac3b5ac7767" gracePeriod=45 Oct 30 13:40:57.105147 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.105115 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" podUID="ee8505de-cac0-4e49-bd15-a044536b10e4" containerName="main" containerID="cri-o://14cab76f1504906ee252c3f7c56a7f92b85f76dab5bed5fbf0bc6dc443be0ab9" gracePeriod=45 Oct 30 13:40:57.107973 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.107946 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" podUID="089cb620-913d-47c7-b8f7-9b2298c194bf" containerName="main" containerID="cri-o://0f23ba70390bbf81ed1106de56905c391ea0ded209bfc6be651ba6be9e4dd4ba" gracePeriod=45 Oct 30 13:40:57.221377 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.221344 4931 generic.go:356] "Generic (PLEG): container finished" podID="f3b342db-9a01-46ad-b476-bfc912d9ce1c" containerID="79e26cbf888e58bce0368a612e9795d8d1056f87ab4e441c8a042ac3b5ac7767" exitCode=0 Oct 30 13:40:57.221503 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.221419 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" event={"ID":"f3b342db-9a01-46ad-b476-bfc912d9ce1c","Type":"ContainerDied","Data":"79e26cbf888e58bce0368a612e9795d8d1056f87ab4e441c8a042ac3b5ac7767"} Oct 30 13:40:57.439550 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.439517 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" Oct 30 13:40:57.443205 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.443179 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" Oct 30 13:40:57.485044 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.485013 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 13:40:57.552761 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.551550 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee8505de-cac0-4e49-bd15-a044536b10e4-home\" (UniqueName: \"kubernetes.io/empty-dir/ee8505de-cac0-4e49-bd15-a044536b10e4-home\") pod \"ee8505de-cac0-4e49-bd15-a044536b10e4\" (UID: \"ee8505de-cac0-4e49-bd15-a044536b10e4\") " Oct 30 13:40:57.552761 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.551620 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee8505de-cac0-4e49-bd15-a044536b10e4-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee8505de-cac0-4e49-bd15-a044536b10e4-tls-certs\") pod \"ee8505de-cac0-4e49-bd15-a044536b10e4\" (UID: \"ee8505de-cac0-4e49-bd15-a044536b10e4\") " Oct 30 13:40:57.552761 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.551653 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b342db-9a01-46ad-b476-bfc912d9ce1c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3b342db-9a01-46ad-b476-bfc912d9ce1c-tmp-dir\") pod \"f3b342db-9a01-46ad-b476-bfc912d9ce1c\" (UID: \"f3b342db-9a01-46ad-b476-bfc912d9ce1c\") " Oct 30 13:40:57.552761 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.551689 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/089cb620-913d-47c7-b8f7-9b2298c194bf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/089cb620-913d-47c7-b8f7-9b2298c194bf-model-cache\") pod \"089cb620-913d-47c7-b8f7-9b2298c194bf\" (UID: \"089cb620-913d-47c7-b8f7-9b2298c194bf\") " Oct 30 13:40:57.552761 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.551715 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b342db-9a01-46ad-b476-bfc912d9ce1c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3b342db-9a01-46ad-b476-bfc912d9ce1c-model-cache\") pod \"f3b342db-9a01-46ad-b476-bfc912d9ce1c\" (UID: \"f3b342db-9a01-46ad-b476-bfc912d9ce1c\") " Oct 30 13:40:57.552761 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.551743 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/089cb620-913d-47c7-b8f7-9b2298c194bf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/089cb620-913d-47c7-b8f7-9b2298c194bf-tmp-dir\") pod \"089cb620-913d-47c7-b8f7-9b2298c194bf\" (UID: \"089cb620-913d-47c7-b8f7-9b2298c194bf\") " Oct 30 13:40:57.552761 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.551773 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee8505de-cac0-4e49-bd15-a044536b10e4-kube-api-access-nmvb4\" (UniqueName: \"kubernetes.io/projected/ee8505de-cac0-4e49-bd15-a044536b10e4-kube-api-access-nmvb4\") pod \"ee8505de-cac0-4e49-bd15-a044536b10e4\" (UID: \"ee8505de-cac0-4e49-bd15-a044536b10e4\") " Oct 30 13:40:57.552761 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.551804 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/089cb620-913d-47c7-b8f7-9b2298c194bf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/089cb620-913d-47c7-b8f7-9b2298c194bf-dshm\") pod \"089cb620-913d-47c7-b8f7-9b2298c194bf\" (UID: \"089cb620-913d-47c7-b8f7-9b2298c194bf\") " Oct 30 13:40:57.552761 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.551832 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b342db-9a01-46ad-b476-bfc912d9ce1c-home\" (UniqueName: \"kubernetes.io/empty-dir/f3b342db-9a01-46ad-b476-bfc912d9ce1c-home\") pod \"f3b342db-9a01-46ad-b476-bfc912d9ce1c\" (UID: \"f3b342db-9a01-46ad-b476-bfc912d9ce1c\") " Oct 30 13:40:57.552761 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.551862 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/089cb620-913d-47c7-b8f7-9b2298c194bf-home\" (UniqueName: \"kubernetes.io/empty-dir/089cb620-913d-47c7-b8f7-9b2298c194bf-home\") pod \"089cb620-913d-47c7-b8f7-9b2298c194bf\" (UID: \"089cb620-913d-47c7-b8f7-9b2298c194bf\") " Oct 30 13:40:57.552761 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.551893 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee8505de-cac0-4e49-bd15-a044536b10e4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee8505de-cac0-4e49-bd15-a044536b10e4-model-cache\") pod \"ee8505de-cac0-4e49-bd15-a044536b10e4\" (UID: \"ee8505de-cac0-4e49-bd15-a044536b10e4\") " Oct 30 13:40:57.552761 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.551944 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3b342db-9a01-46ad-b476-bfc912d9ce1c-tls-certs\" (UniqueName: \"kubernetes.io/secret/f3b342db-9a01-46ad-b476-bfc912d9ce1c-tls-certs\") pod \"f3b342db-9a01-46ad-b476-bfc912d9ce1c\" (UID: \"f3b342db-9a01-46ad-b476-bfc912d9ce1c\") " Oct 30 13:40:57.552761 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.551987 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b342db-9a01-46ad-b476-bfc912d9ce1c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3b342db-9a01-46ad-b476-bfc912d9ce1c-dshm\") pod \"f3b342db-9a01-46ad-b476-bfc912d9ce1c\" (UID: \"f3b342db-9a01-46ad-b476-bfc912d9ce1c\") " Oct 30 13:40:57.552761 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.552014 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee8505de-cac0-4e49-bd15-a044536b10e4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee8505de-cac0-4e49-bd15-a044536b10e4-tmp-dir\") pod \"ee8505de-cac0-4e49-bd15-a044536b10e4\" (UID: \"ee8505de-cac0-4e49-bd15-a044536b10e4\") " Oct 30 13:40:57.552761 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.552073 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b342db-9a01-46ad-b476-bfc912d9ce1c-kube-api-access-pw5v9\" (UniqueName: \"kubernetes.io/projected/f3b342db-9a01-46ad-b476-bfc912d9ce1c-kube-api-access-pw5v9\") pod \"f3b342db-9a01-46ad-b476-bfc912d9ce1c\" (UID: \"f3b342db-9a01-46ad-b476-bfc912d9ce1c\") " Oct 30 13:40:57.552761 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.552141 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee8505de-cac0-4e49-bd15-a044536b10e4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee8505de-cac0-4e49-bd15-a044536b10e4-dshm\") pod \"ee8505de-cac0-4e49-bd15-a044536b10e4\" (UID: \"ee8505de-cac0-4e49-bd15-a044536b10e4\") " Oct 30 13:40:57.552761 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.552152 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/089cb620-913d-47c7-b8f7-9b2298c194bf-model-cache" pod "089cb620-913d-47c7-b8f7-9b2298c194bf" (UID: "089cb620-913d-47c7-b8f7-9b2298c194bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:57.552761 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.552186 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/089cb620-913d-47c7-b8f7-9b2298c194bf-tls-certs\" (UniqueName: \"kubernetes.io/secret/089cb620-913d-47c7-b8f7-9b2298c194bf-tls-certs\") pod \"089cb620-913d-47c7-b8f7-9b2298c194bf\" (UID: \"089cb620-913d-47c7-b8f7-9b2298c194bf\") " Oct 30 13:40:57.552761 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.552215 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/089cb620-913d-47c7-b8f7-9b2298c194bf-kube-api-access-wtjzs\" (UniqueName: \"kubernetes.io/projected/089cb620-913d-47c7-b8f7-9b2298c194bf-kube-api-access-wtjzs\") pod \"089cb620-913d-47c7-b8f7-9b2298c194bf\" (UID: \"089cb620-913d-47c7-b8f7-9b2298c194bf\") " Oct 30 13:40:57.552761 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.552439 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8505de-cac0-4e49-bd15-a044536b10e4-home" pod "ee8505de-cac0-4e49-bd15-a044536b10e4" (UID: "ee8505de-cac0-4e49-bd15-a044536b10e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:57.554064 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.553859 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b342db-9a01-46ad-b476-bfc912d9ce1c-home" pod "f3b342db-9a01-46ad-b476-bfc912d9ce1c" (UID: "f3b342db-9a01-46ad-b476-bfc912d9ce1c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:57.554064 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.553940 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/089cb620-913d-47c7-b8f7-9b2298c194bf-tmp-dir" pod "089cb620-913d-47c7-b8f7-9b2298c194bf" (UID: "089cb620-913d-47c7-b8f7-9b2298c194bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:57.554064 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.554020 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/089cb620-913d-47c7-b8f7-9b2298c194bf-home" pod "089cb620-913d-47c7-b8f7-9b2298c194bf" (UID: "089cb620-913d-47c7-b8f7-9b2298c194bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:57.554184 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.554144 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8505de-cac0-4e49-bd15-a044536b10e4-model-cache" pod "ee8505de-cac0-4e49-bd15-a044536b10e4" (UID: "ee8505de-cac0-4e49-bd15-a044536b10e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:57.555678 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.554413 4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee8505de-cac0-4e49-bd15-a044536b10e4-home\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:57.555678 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.554440 4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/089cb620-913d-47c7-b8f7-9b2298c194bf-model-cache\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:57.555678 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.554489 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b342db-9a01-46ad-b476-bfc912d9ce1c-model-cache" pod "f3b342db-9a01-46ad-b476-bfc912d9ce1c" (UID: "f3b342db-9a01-46ad-b476-bfc912d9ce1c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:57.555678 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.555572 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee8505de-cac0-4e49-bd15-a044536b10e4-tls-certs" pod "ee8505de-cac0-4e49-bd15-a044536b10e4" (UID: "ee8505de-cac0-4e49-bd15-a044536b10e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:40:57.555678 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.555650 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8505de-cac0-4e49-bd15-a044536b10e4-tmp-dir" pod "ee8505de-cac0-4e49-bd15-a044536b10e4" (UID: "ee8505de-cac0-4e49-bd15-a044536b10e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:57.556075 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.555811 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b342db-9a01-46ad-b476-bfc912d9ce1c-tmp-dir" pod "f3b342db-9a01-46ad-b476-bfc912d9ce1c" (UID: "f3b342db-9a01-46ad-b476-bfc912d9ce1c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:57.557622 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.557592 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/089cb620-913d-47c7-b8f7-9b2298c194bf-tls-certs" pod "089cb620-913d-47c7-b8f7-9b2298c194bf" (UID: "089cb620-913d-47c7-b8f7-9b2298c194bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:40:57.558072 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.558034 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3b342db-9a01-46ad-b476-bfc912d9ce1c-tls-certs" pod "f3b342db-9a01-46ad-b476-bfc912d9ce1c" (UID: "f3b342db-9a01-46ad-b476-bfc912d9ce1c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:40:57.559351 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.559323 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b342db-9a01-46ad-b476-bfc912d9ce1c-kube-api-access-pw5v9" pod "f3b342db-9a01-46ad-b476-bfc912d9ce1c" (UID: "f3b342db-9a01-46ad-b476-bfc912d9ce1c"). InnerVolumeSpecName "kube-api-access-pw5v9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:57.559447 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.559367 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee8505de-cac0-4e49-bd15-a044536b10e4-kube-api-access-nmvb4" pod "ee8505de-cac0-4e49-bd15-a044536b10e4" (UID: "ee8505de-cac0-4e49-bd15-a044536b10e4"). InnerVolumeSpecName "kube-api-access-nmvb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:57.560121 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.560096 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/089cb620-913d-47c7-b8f7-9b2298c194bf-dshm" pod "089cb620-913d-47c7-b8f7-9b2298c194bf" (UID: "089cb620-913d-47c7-b8f7-9b2298c194bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:57.560244 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.560168 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b342db-9a01-46ad-b476-bfc912d9ce1c-dshm" pod "f3b342db-9a01-46ad-b476-bfc912d9ce1c" (UID: "f3b342db-9a01-46ad-b476-bfc912d9ce1c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:57.560385 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.560365 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/089cb620-913d-47c7-b8f7-9b2298c194bf-kube-api-access-wtjzs" pod "089cb620-913d-47c7-b8f7-9b2298c194bf" (UID: "089cb620-913d-47c7-b8f7-9b2298c194bf"). InnerVolumeSpecName "kube-api-access-wtjzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:57.561019 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.560997 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8505de-cac0-4e49-bd15-a044536b10e4-dshm" pod "ee8505de-cac0-4e49-bd15-a044536b10e4" (UID: "ee8505de-cac0-4e49-bd15-a044536b10e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:57.655756 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.655456 4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee8505de-cac0-4e49-bd15-a044536b10e4-tls-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:57.655756 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.655499 4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3b342db-9a01-46ad-b476-bfc912d9ce1c-tmp-dir\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:57.655756 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.655514 4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3b342db-9a01-46ad-b476-bfc912d9ce1c-model-cache\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:57.655756 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.655527 4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/089cb620-913d-47c7-b8f7-9b2298c194bf-tmp-dir\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:57.655756 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.655541 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmvb4\" (UniqueName: \"kubernetes.io/projected/ee8505de-cac0-4e49-bd15-a044536b10e4-kube-api-access-nmvb4\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:57.655756 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.655554 4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/089cb620-913d-47c7-b8f7-9b2298c194bf-dshm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:57.655756 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.655567 4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3b342db-9a01-46ad-b476-bfc912d9ce1c-home\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:57.655756 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.655579 4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/089cb620-913d-47c7-b8f7-9b2298c194bf-home\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:57.655756 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.655592 4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee8505de-cac0-4e49-bd15-a044536b10e4-model-cache\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:57.655756 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.655605 4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3b342db-9a01-46ad-b476-bfc912d9ce1c-tls-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:57.655756 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.655617 4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3b342db-9a01-46ad-b476-bfc912d9ce1c-dshm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:57.655756 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.655629 4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee8505de-cac0-4e49-bd15-a044536b10e4-tmp-dir\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:57.655756 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.655642 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw5v9\" (UniqueName: \"kubernetes.io/projected/f3b342db-9a01-46ad-b476-bfc912d9ce1c-kube-api-access-pw5v9\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:57.655756 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.655656 4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee8505de-cac0-4e49-bd15-a044536b10e4-dshm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:57.655756 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.655670 4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/089cb620-913d-47c7-b8f7-9b2298c194bf-tls-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:57.655756 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:57.655684 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtjzs\" (UniqueName: \"kubernetes.io/projected/089cb620-913d-47c7-b8f7-9b2298c194bf-kube-api-access-wtjzs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:58.231426 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.231395 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" Oct 30 13:40:58.231865 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.231424 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" event={"ID":"f3b342db-9a01-46ad-b476-bfc912d9ce1c","Type":"ContainerDied","Data":"96c28132141ca5d05b96760ebec98321de64feca067761e14ac013b3c1c8c664"} Oct 30 13:40:58.231865 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.231470 4931 scope.go:122] "RemoveContainer" containerID="79e26cbf888e58bce0368a612e9795d8d1056f87ab4e441c8a042ac3b5ac7767" Oct 30 13:40:58.232937 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.232898 4931 generic.go:356] "Generic (PLEG): container finished" podID="089cb620-913d-47c7-b8f7-9b2298c194bf" containerID="0f23ba70390bbf81ed1106de56905c391ea0ded209bfc6be651ba6be9e4dd4ba" exitCode=0 Oct 30 13:40:58.233077 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.232951 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" Oct 30 13:40:58.233077 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.232985 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" event={"ID":"089cb620-913d-47c7-b8f7-9b2298c194bf","Type":"ContainerDied","Data":"0f23ba70390bbf81ed1106de56905c391ea0ded209bfc6be651ba6be9e4dd4ba"} Oct 30 13:40:58.233077 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.233021 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj" event={"ID":"089cb620-913d-47c7-b8f7-9b2298c194bf","Type":"ContainerDied","Data":"6f7e1c7bbc7b3fafe096727bd0c94a4dcbf58ced4641102a68a0fe0dbe1a06f9"} Oct 30 13:40:58.234731 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.234706 4931 generic.go:356] "Generic (PLEG): container finished" podID="ee8505de-cac0-4e49-bd15-a044536b10e4" containerID="14cab76f1504906ee252c3f7c56a7f92b85f76dab5bed5fbf0bc6dc443be0ab9" exitCode=0 Oct 30 13:40:58.234844 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.234744 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" Oct 30 13:40:58.234844 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.234763 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" event={"ID":"ee8505de-cac0-4e49-bd15-a044536b10e4","Type":"ContainerDied","Data":"14cab76f1504906ee252c3f7c56a7f92b85f76dab5bed5fbf0bc6dc443be0ab9"} Oct 30 13:40:58.234844 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.234814 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k" event={"ID":"ee8505de-cac0-4e49-bd15-a044536b10e4","Type":"ContainerDied","Data":"b5ffa2b94e368885f63f625aed37e31a8f6518c5c6526887b27f0886935a0b14"} Oct 30 13:40:58.242813 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.242790 4931 scope.go:122] "RemoveContainer" containerID="0f23ba70390bbf81ed1106de56905c391ea0ded209bfc6be651ba6be9e4dd4ba" Oct 30 13:40:58.252880 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.252857 4931 scope.go:122] "RemoveContainer" containerID="0f23ba70390bbf81ed1106de56905c391ea0ded209bfc6be651ba6be9e4dd4ba" Oct 30 13:40:58.253141 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:40:58.253120 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f23ba70390bbf81ed1106de56905c391ea0ded209bfc6be651ba6be9e4dd4ba\": container with ID starting with 0f23ba70390bbf81ed1106de56905c391ea0ded209bfc6be651ba6be9e4dd4ba not found: ID does not exist" containerID="0f23ba70390bbf81ed1106de56905c391ea0ded209bfc6be651ba6be9e4dd4ba" Oct 30 13:40:58.253225 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.253152 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f23ba70390bbf81ed1106de56905c391ea0ded209bfc6be651ba6be9e4dd4ba"} err="failed to get container status \"0f23ba70390bbf81ed1106de56905c391ea0ded209bfc6be651ba6be9e4dd4ba\": rpc error: code = NotFound desc = could not find container \"0f23ba70390bbf81ed1106de56905c391ea0ded209bfc6be651ba6be9e4dd4ba\": container with ID starting with 0f23ba70390bbf81ed1106de56905c391ea0ded209bfc6be651ba6be9e4dd4ba not found: ID does not exist" Oct 30 13:40:58.253225 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.253175 4931 scope.go:122] "RemoveContainer" containerID="14cab76f1504906ee252c3f7c56a7f92b85f76dab5bed5fbf0bc6dc443be0ab9" Oct 30 13:40:58.261660 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.261643 4931 scope.go:122] "RemoveContainer" containerID="14cab76f1504906ee252c3f7c56a7f92b85f76dab5bed5fbf0bc6dc443be0ab9" Oct 30 13:40:58.261921 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:40:58.261895 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14cab76f1504906ee252c3f7c56a7f92b85f76dab5bed5fbf0bc6dc443be0ab9\": container with ID starting with 14cab76f1504906ee252c3f7c56a7f92b85f76dab5bed5fbf0bc6dc443be0ab9 not found: ID does not exist" containerID="14cab76f1504906ee252c3f7c56a7f92b85f76dab5bed5fbf0bc6dc443be0ab9" Oct 30 13:40:58.261962 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.261928 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14cab76f1504906ee252c3f7c56a7f92b85f76dab5bed5fbf0bc6dc443be0ab9"} err="failed to get container status \"14cab76f1504906ee252c3f7c56a7f92b85f76dab5bed5fbf0bc6dc443be0ab9\": rpc error: code = NotFound desc = could not find container \"14cab76f1504906ee252c3f7c56a7f92b85f76dab5bed5fbf0bc6dc443be0ab9\": container with ID starting with 14cab76f1504906ee252c3f7c56a7f92b85f76dab5bed5fbf0bc6dc443be0ab9 not found: ID does not exist" Oct 30 13:40:58.287722 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.287696 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj"] Oct 30 13:40:58.300881 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.300860 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-7a3603/premium-w5-7a3603-kserve-7dcffdbbcc-6t4sj"] Oct 30 13:40:58.328603 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.328574 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz"] Oct 30 13:40:58.332464 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.332441 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz"] Oct 30 13:40:58.371856 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.371782 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/distinct-w5-7a3603-kserve-f7997fc8d-n9bfz" podUID="f3b342db-9a01-46ad-b476-bfc912d9ce1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:40:58.392626 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.392594 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k"] Oct 30 13:40:58.397912 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.397881 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-7a3603/distinct-2-w5-7a3603-kserve-ff956599f-m5t5k"] Oct 30 13:40:58.725607 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.725582 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" Oct 30 13:40:58.868784 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.868744 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc65620b-073b-4e13-82da-38526fb90da3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc65620b-073b-4e13-82da-38526fb90da3-workload-socket\") pod \"fc65620b-073b-4e13-82da-38526fb90da3\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " Oct 30 13:40:58.868784 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.868815 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc65620b-073b-4e13-82da-38526fb90da3-istio-token\" (UniqueName: \"kubernetes.io/projected/fc65620b-073b-4e13-82da-38526fb90da3-istio-token\") pod \"fc65620b-073b-4e13-82da-38526fb90da3\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " Oct 30 13:40:58.869150 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.868847 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc65620b-073b-4e13-82da-38526fb90da3-kube-api-access-8lm54\" (UniqueName: \"kubernetes.io/projected/fc65620b-073b-4e13-82da-38526fb90da3-kube-api-access-8lm54\") pod \"fc65620b-073b-4e13-82da-38526fb90da3\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " Oct 30 13:40:58.869150 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.868905 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc65620b-073b-4e13-82da-38526fb90da3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc65620b-073b-4e13-82da-38526fb90da3-istiod-ca-cert\") pod \"fc65620b-073b-4e13-82da-38526fb90da3\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " Oct 30 13:40:58.869150 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.868975 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc65620b-073b-4e13-82da-38526fb90da3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc65620b-073b-4e13-82da-38526fb90da3-istio-envoy\") pod \"fc65620b-073b-4e13-82da-38526fb90da3\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " Oct 30 13:40:58.869150 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.869029 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc65620b-073b-4e13-82da-38526fb90da3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc65620b-073b-4e13-82da-38526fb90da3-workload-certs\") pod \"fc65620b-073b-4e13-82da-38526fb90da3\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " Oct 30 13:40:58.869150 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.869128 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc65620b-073b-4e13-82da-38526fb90da3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc65620b-073b-4e13-82da-38526fb90da3-istio-podinfo\") pod \"fc65620b-073b-4e13-82da-38526fb90da3\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " Oct 30 13:40:58.869554 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.869156 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc65620b-073b-4e13-82da-38526fb90da3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc65620b-073b-4e13-82da-38526fb90da3-credential-socket\") pod \"fc65620b-073b-4e13-82da-38526fb90da3\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " Oct 30 13:40:58.869554 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.869213 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc65620b-073b-4e13-82da-38526fb90da3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc65620b-073b-4e13-82da-38526fb90da3-istio-data\") pod \"fc65620b-073b-4e13-82da-38526fb90da3\" (UID: \"fc65620b-073b-4e13-82da-38526fb90da3\") " Oct 30 13:40:58.869795 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.869767 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc65620b-073b-4e13-82da-38526fb90da3-istio-data" pod "fc65620b-073b-4e13-82da-38526fb90da3" (UID: "fc65620b-073b-4e13-82da-38526fb90da3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:58.870302 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.870244 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc65620b-073b-4e13-82da-38526fb90da3-workload-socket" pod "fc65620b-073b-4e13-82da-38526fb90da3" (UID: "fc65620b-073b-4e13-82da-38526fb90da3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:58.870506 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.870445 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc65620b-073b-4e13-82da-38526fb90da3-workload-certs" pod "fc65620b-073b-4e13-82da-38526fb90da3" (UID: "fc65620b-073b-4e13-82da-38526fb90da3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:58.870617 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.870594 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc65620b-073b-4e13-82da-38526fb90da3-istiod-ca-cert" pod "fc65620b-073b-4e13-82da-38526fb90da3" (UID: "fc65620b-073b-4e13-82da-38526fb90da3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:40:58.870736 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.870712 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc65620b-073b-4e13-82da-38526fb90da3-credential-socket" pod "fc65620b-073b-4e13-82da-38526fb90da3" (UID: "fc65620b-073b-4e13-82da-38526fb90da3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:58.874801 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.874757 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc65620b-073b-4e13-82da-38526fb90da3-istio-token" pod "fc65620b-073b-4e13-82da-38526fb90da3" (UID: "fc65620b-073b-4e13-82da-38526fb90da3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:58.876139 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.876099 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc65620b-073b-4e13-82da-38526fb90da3-istio-envoy" pod "fc65620b-073b-4e13-82da-38526fb90da3" (UID: "fc65620b-073b-4e13-82da-38526fb90da3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:58.876397 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.876374 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc65620b-073b-4e13-82da-38526fb90da3-istio-podinfo" pod "fc65620b-073b-4e13-82da-38526fb90da3" (UID: "fc65620b-073b-4e13-82da-38526fb90da3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:40:58.881212 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.881176 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc65620b-073b-4e13-82da-38526fb90da3-kube-api-access-8lm54" pod "fc65620b-073b-4e13-82da-38526fb90da3" (UID: "fc65620b-073b-4e13-82da-38526fb90da3"). InnerVolumeSpecName "kube-api-access-8lm54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:58.970392 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.970350 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc65620b-073b-4e13-82da-38526fb90da3-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:58.970392 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.970384 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc65620b-073b-4e13-82da-38526fb90da3-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:58.970392 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.970394 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc65620b-073b-4e13-82da-38526fb90da3-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:58.970646 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.970404 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc65620b-073b-4e13-82da-38526fb90da3-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:58.970646 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.970413 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc65620b-073b-4e13-82da-38526fb90da3-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:58.970646 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.970422 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc65620b-073b-4e13-82da-38526fb90da3-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:58.970646 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.970430 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lm54\" (UniqueName: \"kubernetes.io/projected/fc65620b-073b-4e13-82da-38526fb90da3-kube-api-access-8lm54\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:58.970646 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.970438 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc65620b-073b-4e13-82da-38526fb90da3-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:58.970646 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:58.970446 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc65620b-073b-4e13-82da-38526fb90da3-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:40:59.243654 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:59.243569 4931 generic.go:356] "Generic (PLEG): container finished" podID="fc65620b-073b-4e13-82da-38526fb90da3" containerID="decf2dabff7f6ac00617d6984a38c747dafc1a911633f93c4f2b5cef67483bae" exitCode=0 Oct 30 13:40:59.243654 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:59.243640 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" Oct 30 13:40:59.244201 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:59.243656 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" event={"ID":"fc65620b-073b-4e13-82da-38526fb90da3","Type":"ContainerDied","Data":"decf2dabff7f6ac00617d6984a38c747dafc1a911633f93c4f2b5cef67483bae"} Oct 30 13:40:59.244201 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:59.243745 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8" event={"ID":"fc65620b-073b-4e13-82da-38526fb90da3","Type":"ContainerDied","Data":"e972492a39d424bd03f26bee52131d022e19afd40ac6d4722f666ee464de5f85"} Oct 30 13:40:59.244201 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:59.243769 4931 scope.go:122] "RemoveContainer" containerID="decf2dabff7f6ac00617d6984a38c747dafc1a911633f93c4f2b5cef67483bae" Oct 30 13:40:59.254428 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:59.254409 4931 scope.go:122] "RemoveContainer" containerID="decf2dabff7f6ac00617d6984a38c747dafc1a911633f93c4f2b5cef67483bae" Oct 30 13:40:59.254685 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:40:59.254665 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"decf2dabff7f6ac00617d6984a38c747dafc1a911633f93c4f2b5cef67483bae\": container with ID starting with decf2dabff7f6ac00617d6984a38c747dafc1a911633f93c4f2b5cef67483bae not found: ID does not exist" containerID="decf2dabff7f6ac00617d6984a38c747dafc1a911633f93c4f2b5cef67483bae" Oct 30 13:40:59.254733 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:59.254693 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"decf2dabff7f6ac00617d6984a38c747dafc1a911633f93c4f2b5cef67483bae"} err="failed to get container status \"decf2dabff7f6ac00617d6984a38c747dafc1a911633f93c4f2b5cef67483bae\": rpc error: code = NotFound desc = could not find container \"decf2dabff7f6ac00617d6984a38c747dafc1a911633f93c4f2b5cef67483bae\": container with ID starting with decf2dabff7f6ac00617d6984a38c747dafc1a911633f93c4f2b5cef67483bae not found: ID does not exist" Oct 30 13:40:59.300365 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:59.300333 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8"] Oct 30 13:40:59.304326 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:59.304304 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8"] Oct 30 13:40:59.512836 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:59.512759 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="089cb620-913d-47c7-b8f7-9b2298c194bf" path="/var/lib/kubelet/pods/089cb620-913d-47c7-b8f7-9b2298c194bf/volumes" Oct 30 13:40:59.513169 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:59.513156 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee8505de-cac0-4e49-bd15-a044536b10e4" path="/var/lib/kubelet/pods/ee8505de-cac0-4e49-bd15-a044536b10e4/volumes" Oct 30 13:40:59.513630 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:59.513605 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3b342db-9a01-46ad-b476-bfc912d9ce1c" path="/var/lib/kubelet/pods/f3b342db-9a01-46ad-b476-bfc912d9ce1c/volumes" Oct 30 13:40:59.514760 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:59.514716 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc65620b-073b-4e13-82da-38526fb90da3" path="/var/lib/kubelet/pods/fc65620b-073b-4e13-82da-38526fb90da3/volumes" Oct 30 13:40:59.622550 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:59.622515 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd90cb-bzxnl"] Oct 30 13:40:59.624883 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:40:59.624858 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd90cb-bzxnl"] Oct 30 13:41:01.513561 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:01.513527 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a00926d5-b69b-4a8a-b076-68ad63208f30" path="/var/lib/kubelet/pods/a00926d5-b69b-4a8a-b076-68ad63208f30/volumes" Oct 30 13:41:01.970835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:01.970796 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e1b470-jqbrm"] Oct 30 13:41:01.971512 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:01.971490 4931 cpu_manager.go:414] "containerMap: removing container" podUID="089cb620-913d-47c7-b8f7-9b2298c194bf" containerName="main" Oct 30 13:41:01.971563 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:01.971516 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="089cb620-913d-47c7-b8f7-9b2298c194bf" containerName="main" Oct 30 13:41:01.971563 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:01.971543 4931 cpu_manager.go:414] "containerMap: removing container" podUID="e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" containerName="main" Oct 30 13:41:01.971563 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:01.971551 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" containerName="main" Oct 30 13:41:01.971671 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:01.971593 4931 cpu_manager.go:414] "containerMap: removing container" podUID="f3b342db-9a01-46ad-b476-bfc912d9ce1c" containerName="main" Oct 30 13:41:01.971671 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:01.971602 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b342db-9a01-46ad-b476-bfc912d9ce1c" containerName="main" Oct 30 13:41:01.971671 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:01.971613 4931 cpu_manager.go:414] "containerMap: removing container" podUID="ee8505de-cac0-4e49-bd15-a044536b10e4" containerName="main" Oct 30 13:41:01.971671 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:01.971621 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee8505de-cac0-4e49-bd15-a044536b10e4" containerName="main" Oct 30 13:41:01.971671 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:01.971638 4931 cpu_manager.go:414] "containerMap: removing container" podUID="03309fcc-c2dd-4424-9deb-c34596cd1558" containerName="payload-processing" Oct 30 13:41:01.971671 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:01.971646 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03309fcc-c2dd-4424-9deb-c34596cd1558" containerName="payload-processing" Oct 30 13:41:01.971671 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:01.971658 4931 cpu_manager.go:414] "containerMap: removing container" podUID="7ad1989a-c73f-4647-9e50-855605bdb08e" containerName="main" Oct 30 13:41:01.971671 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:01.971665 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ad1989a-c73f-4647-9e50-855605bdb08e" containerName="main" Oct 30 13:41:01.971922 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:01.971676 4931 cpu_manager.go:414] "containerMap: removing container" podUID="606b252b-881c-4d40-8ba9-fc4ed937a995" containerName="main" Oct 30 13:41:01.971922 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:01.971683 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="606b252b-881c-4d40-8ba9-fc4ed937a995" containerName="main" Oct 30 13:41:01.971922 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:01.971694 4931 cpu_manager.go:414] "containerMap: removing container" podUID="fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9" containerName="istio-proxy" Oct 30 13:41:01.971922 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:01.971702 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9" containerName="istio-proxy" Oct 30 13:41:01.971922 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:01.971714 4931 cpu_manager.go:414] "containerMap: removing container" podUID="f9e56860-9840-4bd5-bf11-104f28578afe" containerName="revoke-keys" Oct 30 13:41:01.971922 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:01.971721 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9e56860-9840-4bd5-bf11-104f28578afe" containerName="revoke-keys" Oct 30 13:41:01.971922 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:01.971732 4931 cpu_manager.go:414] "containerMap: removing container" podUID="fc65620b-073b-4e13-82da-38526fb90da3" containerName="istio-proxy" Oct 30 13:41:01.971922 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:01.971740 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc65620b-073b-4e13-82da-38526fb90da3" containerName="istio-proxy" Oct 30 13:41:01.971922 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:01.971840 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b342db-9a01-46ad-b476-bfc912d9ce1c" containerName="main" Oct 30 13:41:01.971922 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:01.971851 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee8505de-cac0-4e49-bd15-a044536b10e4" containerName="main" Oct 30 13:41:01.971922 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:01.971896 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="03309fcc-c2dd-4424-9deb-c34596cd1558" containerName="payload-processing" Oct 30 13:41:01.972314 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:01.971908 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ad1989a-c73f-4647-9e50-855605bdb08e" containerName="main" Oct 30 13:41:01.972314 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:01.971941 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="606b252b-881c-4d40-8ba9-fc4ed937a995" containerName="main" Oct 30 13:41:01.972314 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:01.971953 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcbc4d0a-6516-4cc5-ab8a-0e19a36e3ae9" containerName="istio-proxy" Oct 30 13:41:01.972314 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:01.971962 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9e56860-9840-4bd5-bf11-104f28578afe" containerName="revoke-keys" Oct 30 13:41:01.972314 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:01.971971 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc65620b-073b-4e13-82da-38526fb90da3" containerName="istio-proxy" Oct 30 13:41:01.972314 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:01.971982 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="089cb620-913d-47c7-b8f7-9b2298c194bf" containerName="main" Oct 30 13:41:01.972314 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:01.971992 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="e52bb4f7-ceff-4e39-91ce-a8f48b205a0a" containerName="main" Oct 30 13:41:01.975565 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:01.975542 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e1b470-jqbrm" Oct 30 13:41:02.002615 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:02.002589 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e1b470-jqbrm"] Oct 30 13:41:02.046963 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:02.046926 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:41:02.047221 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:02.047140 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:41:02.100292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:02.100236 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwztt\" (UniqueName: \"kubernetes.io/projected/fb8000b8-760a-42bb-9c19-6fdc97354a6a-kube-api-access-qwztt\") pod \"maas-api-revoke-sub-c1e1b470-jqbrm\" (UID: \"fb8000b8-760a-42bb-9c19-6fdc97354a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e1b470-jqbrm" Oct 30 13:41:02.100448 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:02.100341 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb8000b8-760a-42bb-9c19-6fdc97354a6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e1b470-jqbrm\" (UID: \"fb8000b8-760a-42bb-9c19-6fdc97354a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e1b470-jqbrm" Oct 30 13:41:02.100448 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:02.100417 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb8000b8-760a-42bb-9c19-6fdc97354a6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e1b470-jqbrm\" (UID: \"fb8000b8-760a-42bb-9c19-6fdc97354a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e1b470-jqbrm" Oct 30 13:41:02.201858 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:02.201826 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwztt\" (UniqueName: \"kubernetes.io/projected/fb8000b8-760a-42bb-9c19-6fdc97354a6a-kube-api-access-qwztt\") pod \"maas-api-revoke-sub-c1e1b470-jqbrm\" (UID: \"fb8000b8-760a-42bb-9c19-6fdc97354a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e1b470-jqbrm" Oct 30 13:41:02.202010 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:02.201878 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb8000b8-760a-42bb-9c19-6fdc97354a6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e1b470-jqbrm\" (UID: \"fb8000b8-760a-42bb-9c19-6fdc97354a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e1b470-jqbrm" Oct 30 13:41:02.202010 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:02.201941 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb8000b8-760a-42bb-9c19-6fdc97354a6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e1b470-jqbrm\" (UID: \"fb8000b8-760a-42bb-9c19-6fdc97354a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e1b470-jqbrm" Oct 30 13:41:02.202668 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:02.202648 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb8000b8-760a-42bb-9c19-6fdc97354a6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e1b470-jqbrm\" (UID: \"fb8000b8-760a-42bb-9c19-6fdc97354a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e1b470-jqbrm" Oct 30 13:41:02.204192 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:02.204172 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb8000b8-760a-42bb-9c19-6fdc97354a6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e1b470-jqbrm\" (UID: \"fb8000b8-760a-42bb-9c19-6fdc97354a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e1b470-jqbrm" Oct 30 13:41:02.229171 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:02.229102 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwztt\" (UniqueName: \"kubernetes.io/projected/fb8000b8-760a-42bb-9c19-6fdc97354a6a-kube-api-access-qwztt\") pod \"maas-api-revoke-sub-c1e1b470-jqbrm\" (UID: \"fb8000b8-760a-42bb-9c19-6fdc97354a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e1b470-jqbrm" Oct 30 13:41:02.289396 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:02.289365 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e1b470-jqbrm" Oct 30 13:41:02.596548 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:02.596521 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e1b470-jqbrm"] Oct 30 13:41:02.597165 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:41:02.597137 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb8000b8_760a_42bb_9c19_6fdc97354a6a.slice/crio-d99535ba7b84f88298aaf55747708ce23c449ad80d428727c526669fb1e067e4 WatchSource:0}: Error finding container d99535ba7b84f88298aaf55747708ce23c449ad80d428727c526669fb1e067e4: Status 404 returned error can't find the container with id d99535ba7b84f88298aaf55747708ce23c449ad80d428727c526669fb1e067e4 Oct 30 13:41:03.278659 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:03.278591 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e1b470-jqbrm" event={"ID":"fb8000b8-760a-42bb-9c19-6fdc97354a6a","Type":"ContainerStarted","Data":"16c22459193205128ca577a25a1a21b68860060607bf11eb6a12dbebbc463b7a"} Oct 30 13:41:03.278961 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:03.278690 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e1b470-jqbrm" event={"ID":"fb8000b8-760a-42bb-9c19-6fdc97354a6a","Type":"ContainerStarted","Data":"d99535ba7b84f88298aaf55747708ce23c449ad80d428727c526669fb1e067e4"} Oct 30 13:41:03.363323 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:03.363253 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e1b470-jqbrm" podStartSLOduration=2.363237642 podStartE2EDuration="2.363237642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:41:03.359958138 +0000 UTC m=+3361.963456584" watchObservedRunningTime="2026-10-30 13:41:03.363237642 +0000 UTC m=+3361.966736087" Oct 30 13:41:03.367722 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:03.367693 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb" podUID="440d993b-e62e-4284-955d-2e79adccec2e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:41:04.286762 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:04.286723 4931 generic.go:356] "Generic (PLEG): container finished" podID="fb8000b8-760a-42bb-9c19-6fdc97354a6a" containerID="16c22459193205128ca577a25a1a21b68860060607bf11eb6a12dbebbc463b7a" exitCode=0 Oct 30 13:41:04.287214 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:04.286808 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e1b470-jqbrm" event={"ID":"fb8000b8-760a-42bb-9c19-6fdc97354a6a","Type":"ContainerDied","Data":"16c22459193205128ca577a25a1a21b68860060607bf11eb6a12dbebbc463b7a"} Oct 30 13:41:04.580450 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:04.580375 4931 scope.go:122] "RemoveContainer" containerID="ff901eab67367b166da25cdddbf18beb67acbc1c54924922c824008bce2c248e" Oct 30 13:41:05.440971 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:05.440941 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e1b470-jqbrm" Oct 30 13:41:05.533515 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:05.533486 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb8000b8-760a-42bb-9c19-6fdc97354a6a-kube-api-access-qwztt\" (UniqueName: \"kubernetes.io/projected/fb8000b8-760a-42bb-9c19-6fdc97354a6a-kube-api-access-qwztt\") pod \"fb8000b8-760a-42bb-9c19-6fdc97354a6a\" (UID: \"fb8000b8-760a-42bb-9c19-6fdc97354a6a\") " Oct 30 13:41:05.533716 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:05.533545 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb8000b8-760a-42bb-9c19-6fdc97354a6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb8000b8-760a-42bb-9c19-6fdc97354a6a-maas-api-xks-ca\") pod \"fb8000b8-760a-42bb-9c19-6fdc97354a6a\" (UID: \"fb8000b8-760a-42bb-9c19-6fdc97354a6a\") " Oct 30 13:41:05.533716 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:05.533673 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb8000b8-760a-42bb-9c19-6fdc97354a6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb8000b8-760a-42bb-9c19-6fdc97354a6a-maas-api-openshift-ca\") pod \"fb8000b8-760a-42bb-9c19-6fdc97354a6a\" (UID: \"fb8000b8-760a-42bb-9c19-6fdc97354a6a\") " Oct 30 13:41:05.534016 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:05.533991 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb8000b8-760a-42bb-9c19-6fdc97354a6a-maas-api-openshift-ca" pod "fb8000b8-760a-42bb-9c19-6fdc97354a6a" (UID: "fb8000b8-760a-42bb-9c19-6fdc97354a6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:41:05.535716 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:05.535688 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb8000b8-760a-42bb-9c19-6fdc97354a6a-kube-api-access-qwztt" pod "fb8000b8-760a-42bb-9c19-6fdc97354a6a" (UID: "fb8000b8-760a-42bb-9c19-6fdc97354a6a"). InnerVolumeSpecName "kube-api-access-qwztt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:05.535828 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:05.535758 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb8000b8-760a-42bb-9c19-6fdc97354a6a-maas-api-xks-ca" pod "fb8000b8-760a-42bb-9c19-6fdc97354a6a" (UID: "fb8000b8-760a-42bb-9c19-6fdc97354a6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:41:05.635059 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:05.634998 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb8000b8-760a-42bb-9c19-6fdc97354a6a-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:41:05.635059 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:05.635022 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwztt\" (UniqueName: \"kubernetes.io/projected/fb8000b8-760a-42bb-9c19-6fdc97354a6a-kube-api-access-qwztt\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:41:05.635059 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:05.635032 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb8000b8-760a-42bb-9c19-6fdc97354a6a-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:41:06.301259 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:06.301221 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e1b470-jqbrm" Oct 30 13:41:06.301471 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:06.301278 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e1b470-jqbrm" event={"ID":"fb8000b8-760a-42bb-9c19-6fdc97354a6a","Type":"ContainerDied","Data":"d99535ba7b84f88298aaf55747708ce23c449ad80d428727c526669fb1e067e4"} Oct 30 13:41:06.301471 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:06.301339 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d99535ba7b84f88298aaf55747708ce23c449ad80d428727c526669fb1e067e4" Oct 30 13:41:06.809679 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:06.809623 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb" Oct 30 13:41:06.949427 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:06.949385 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/440d993b-e62e-4284-955d-2e79adccec2e-kube-api-access-4l2x5\" (UniqueName: \"kubernetes.io/projected/440d993b-e62e-4284-955d-2e79adccec2e-kube-api-access-4l2x5\") pod \"440d993b-e62e-4284-955d-2e79adccec2e\" (UID: \"440d993b-e62e-4284-955d-2e79adccec2e\") " Oct 30 13:41:06.949655 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:06.949480 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/440d993b-e62e-4284-955d-2e79adccec2e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/440d993b-e62e-4284-955d-2e79adccec2e-plugins-config-volume\") pod \"440d993b-e62e-4284-955d-2e79adccec2e\" (UID: \"440d993b-e62e-4284-955d-2e79adccec2e\") " Oct 30 13:41:06.950290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:06.950209 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440d993b-e62e-4284-955d-2e79adccec2e-plugins-config-volume" pod "440d993b-e62e-4284-955d-2e79adccec2e" (UID: "440d993b-e62e-4284-955d-2e79adccec2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:41:06.960798 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:06.960765 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440d993b-e62e-4284-955d-2e79adccec2e-kube-api-access-4l2x5" pod "440d993b-e62e-4284-955d-2e79adccec2e" (UID: "440d993b-e62e-4284-955d-2e79adccec2e"). InnerVolumeSpecName "kube-api-access-4l2x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:07.050990 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:07.050946 4931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/440d993b-e62e-4284-955d-2e79adccec2e-plugins-config-volume\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:41:07.050990 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:07.050983 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l2x5\" (UniqueName: \"kubernetes.io/projected/440d993b-e62e-4284-955d-2e79adccec2e-kube-api-access-4l2x5\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:41:07.313386 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:07.313348 4931 generic.go:356] "Generic (PLEG): container finished" podID="440d993b-e62e-4284-955d-2e79adccec2e" containerID="f437a14b7e3a5be4eb14ad1f26a65ad40adbaa59dd34f6b1a5823245b7081ce2" exitCode=0 Oct 30 13:41:07.313584 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:07.313400 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb" Oct 30 13:41:07.313584 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:07.313434 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb" event={"ID":"440d993b-e62e-4284-955d-2e79adccec2e","Type":"ContainerDied","Data":"f437a14b7e3a5be4eb14ad1f26a65ad40adbaa59dd34f6b1a5823245b7081ce2"} Oct 30 13:41:07.313584 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:07.313502 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb" event={"ID":"440d993b-e62e-4284-955d-2e79adccec2e","Type":"ContainerDied","Data":"f8dd40c4b2f79b0b132b5e8538d6d1e2b11aa3afc1073b049c5197e966cfd0f4"} Oct 30 13:41:07.313584 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:07.313533 4931 scope.go:122] "RemoveContainer" containerID="f437a14b7e3a5be4eb14ad1f26a65ad40adbaa59dd34f6b1a5823245b7081ce2" Oct 30 13:41:07.328120 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:07.328099 4931 scope.go:122] "RemoveContainer" containerID="f437a14b7e3a5be4eb14ad1f26a65ad40adbaa59dd34f6b1a5823245b7081ce2" Oct 30 13:41:07.328514 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:41:07.328480 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f437a14b7e3a5be4eb14ad1f26a65ad40adbaa59dd34f6b1a5823245b7081ce2\": container with ID starting with f437a14b7e3a5be4eb14ad1f26a65ad40adbaa59dd34f6b1a5823245b7081ce2 not found: ID does not exist" containerID="f437a14b7e3a5be4eb14ad1f26a65ad40adbaa59dd34f6b1a5823245b7081ce2" Oct 30 13:41:07.328648 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:07.328525 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f437a14b7e3a5be4eb14ad1f26a65ad40adbaa59dd34f6b1a5823245b7081ce2"} err="failed to get container status \"f437a14b7e3a5be4eb14ad1f26a65ad40adbaa59dd34f6b1a5823245b7081ce2\": rpc error: code = NotFound desc = could not find container \"f437a14b7e3a5be4eb14ad1f26a65ad40adbaa59dd34f6b1a5823245b7081ce2\": container with ID starting with f437a14b7e3a5be4eb14ad1f26a65ad40adbaa59dd34f6b1a5823245b7081ce2 not found: ID does not exist" Oct 30 13:41:07.355136 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:41:07.355100 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:41:07.374245 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:07.374218 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb"] Oct 30 13:41:07.391873 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:07.391844 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b"] Oct 30 13:41:07.392060 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:07.392033 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" podUID="e174882a-3e74-4174-99cf-dcf265fb86f6" containerName="istio-proxy" containerID="cri-o://34bb6dd9415332a7ff06bd21408e100f0b433bfff710c7807d0bbe230e5daf94" gracePeriod=30 Oct 30 13:41:07.402914 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:07.402888 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb"] Oct 30 13:41:07.512573 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:07.512539 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="440d993b-e62e-4284-955d-2e79adccec2e" path="/var/lib/kubelet/pods/440d993b-e62e-4284-955d-2e79adccec2e/volumes" Oct 30 13:41:12.234751 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:12.234711 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:41:12.648170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:12.648145 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" Oct 30 13:41:12.809908 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:12.809810 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e174882a-3e74-4174-99cf-dcf265fb86f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e174882a-3e74-4174-99cf-dcf265fb86f6-istio-envoy\") pod \"e174882a-3e74-4174-99cf-dcf265fb86f6\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " Oct 30 13:41:12.809908 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:12.809873 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e174882a-3e74-4174-99cf-dcf265fb86f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e174882a-3e74-4174-99cf-dcf265fb86f6-credential-socket\") pod \"e174882a-3e74-4174-99cf-dcf265fb86f6\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " Oct 30 13:41:12.809908 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:12.809904 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e174882a-3e74-4174-99cf-dcf265fb86f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e174882a-3e74-4174-99cf-dcf265fb86f6-workload-certs\") pod \"e174882a-3e74-4174-99cf-dcf265fb86f6\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " Oct 30 13:41:12.810210 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:12.809935 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e174882a-3e74-4174-99cf-dcf265fb86f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e174882a-3e74-4174-99cf-dcf265fb86f6-istio-podinfo\") pod \"e174882a-3e74-4174-99cf-dcf265fb86f6\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " Oct 30 13:41:12.810210 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:12.809954 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e174882a-3e74-4174-99cf-dcf265fb86f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e174882a-3e74-4174-99cf-dcf265fb86f6-istiod-ca-cert\") pod \"e174882a-3e74-4174-99cf-dcf265fb86f6\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " Oct 30 13:41:12.810210 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:12.809994 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e174882a-3e74-4174-99cf-dcf265fb86f6-kube-api-access-c86qw\" (UniqueName: \"kubernetes.io/projected/e174882a-3e74-4174-99cf-dcf265fb86f6-kube-api-access-c86qw\") pod \"e174882a-3e74-4174-99cf-dcf265fb86f6\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " Oct 30 13:41:12.810210 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:12.810040 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e174882a-3e74-4174-99cf-dcf265fb86f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e174882a-3e74-4174-99cf-dcf265fb86f6-istio-data\") pod \"e174882a-3e74-4174-99cf-dcf265fb86f6\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " Oct 30 13:41:12.810210 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:12.810065 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e174882a-3e74-4174-99cf-dcf265fb86f6-istio-token\" (UniqueName: \"kubernetes.io/projected/e174882a-3e74-4174-99cf-dcf265fb86f6-istio-token\") pod \"e174882a-3e74-4174-99cf-dcf265fb86f6\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " Oct 30 13:41:12.810210 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:12.810113 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e174882a-3e74-4174-99cf-dcf265fb86f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e174882a-3e74-4174-99cf-dcf265fb86f6-workload-socket\") pod \"e174882a-3e74-4174-99cf-dcf265fb86f6\" (UID: \"e174882a-3e74-4174-99cf-dcf265fb86f6\") " Oct 30 13:41:12.810564 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:12.810205 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e174882a-3e74-4174-99cf-dcf265fb86f6-workload-certs" pod "e174882a-3e74-4174-99cf-dcf265fb86f6" (UID: "e174882a-3e74-4174-99cf-dcf265fb86f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:12.810564 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:12.810217 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e174882a-3e74-4174-99cf-dcf265fb86f6-credential-socket" pod "e174882a-3e74-4174-99cf-dcf265fb86f6" (UID: "e174882a-3e74-4174-99cf-dcf265fb86f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:12.810564 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:12.810421 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e174882a-3e74-4174-99cf-dcf265fb86f6-istiod-ca-cert" pod "e174882a-3e74-4174-99cf-dcf265fb86f6" (UID: "e174882a-3e74-4174-99cf-dcf265fb86f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:41:12.810564 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:12.810490 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e174882a-3e74-4174-99cf-dcf265fb86f6-istio-data" pod "e174882a-3e74-4174-99cf-dcf265fb86f6" (UID: "e174882a-3e74-4174-99cf-dcf265fb86f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:12.810564 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:12.810542 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e174882a-3e74-4174-99cf-dcf265fb86f6-workload-socket" pod "e174882a-3e74-4174-99cf-dcf265fb86f6" (UID: "e174882a-3e74-4174-99cf-dcf265fb86f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:12.810564 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:12.810554 4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e174882a-3e74-4174-99cf-dcf265fb86f6-credential-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:41:12.810564 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:12.810569 4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e174882a-3e74-4174-99cf-dcf265fb86f6-workload-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:41:12.812235 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:12.812209 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e174882a-3e74-4174-99cf-dcf265fb86f6-istio-envoy" pod "e174882a-3e74-4174-99cf-dcf265fb86f6" (UID: "e174882a-3e74-4174-99cf-dcf265fb86f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:12.812392 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:12.812318 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e174882a-3e74-4174-99cf-dcf265fb86f6-istio-podinfo" pod "e174882a-3e74-4174-99cf-dcf265fb86f6" (UID: "e174882a-3e74-4174-99cf-dcf265fb86f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:41:12.812392 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:12.812381 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e174882a-3e74-4174-99cf-dcf265fb86f6-istio-token" pod "e174882a-3e74-4174-99cf-dcf265fb86f6" (UID: "e174882a-3e74-4174-99cf-dcf265fb86f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:12.812497 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:12.812476 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e174882a-3e74-4174-99cf-dcf265fb86f6-kube-api-access-c86qw" pod "e174882a-3e74-4174-99cf-dcf265fb86f6" (UID: "e174882a-3e74-4174-99cf-dcf265fb86f6"). InnerVolumeSpecName "kube-api-access-c86qw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:12.911618 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:12.911551 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c86qw\" (UniqueName: \"kubernetes.io/projected/e174882a-3e74-4174-99cf-dcf265fb86f6-kube-api-access-c86qw\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:41:12.911618 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:12.911578 4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e174882a-3e74-4174-99cf-dcf265fb86f6-istio-data\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:41:12.911618 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:12.911588 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e174882a-3e74-4174-99cf-dcf265fb86f6-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:41:12.911618 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:12.911598 4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e174882a-3e74-4174-99cf-dcf265fb86f6-workload-socket\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:41:12.911618 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:12.911607 4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e174882a-3e74-4174-99cf-dcf265fb86f6-istio-envoy\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:41:12.911618 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:12.911615 4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e174882a-3e74-4174-99cf-dcf265fb86f6-istio-podinfo\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:41:12.911618 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:12.911623 4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e174882a-3e74-4174-99cf-dcf265fb86f6-istiod-ca-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:41:13.355217 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:13.355177 4931 generic.go:356] "Generic (PLEG): container finished" podID="e174882a-3e74-4174-99cf-dcf265fb86f6" containerID="34bb6dd9415332a7ff06bd21408e100f0b433bfff710c7807d0bbe230e5daf94" exitCode=0 Oct 30 13:41:13.355714 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:13.355227 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" Oct 30 13:41:13.355714 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:13.355289 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" event={"ID":"e174882a-3e74-4174-99cf-dcf265fb86f6","Type":"ContainerDied","Data":"34bb6dd9415332a7ff06bd21408e100f0b433bfff710c7807d0bbe230e5daf94"} Oct 30 13:41:13.355714 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:13.355365 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b" event={"ID":"e174882a-3e74-4174-99cf-dcf265fb86f6","Type":"ContainerDied","Data":"25839ecf684dd1c16888c423eda7aab3af86efa181b52687b67f71531b71a133"} Oct 30 13:41:13.355714 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:13.355389 4931 scope.go:122] "RemoveContainer" containerID="34bb6dd9415332a7ff06bd21408e100f0b433bfff710c7807d0bbe230e5daf94" Oct 30 13:41:13.365720 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:13.365700 4931 scope.go:122] "RemoveContainer" containerID="34bb6dd9415332a7ff06bd21408e100f0b433bfff710c7807d0bbe230e5daf94" Oct 30 13:41:13.365999 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:41:13.365976 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34bb6dd9415332a7ff06bd21408e100f0b433bfff710c7807d0bbe230e5daf94\": container with ID starting with 34bb6dd9415332a7ff06bd21408e100f0b433bfff710c7807d0bbe230e5daf94 not found: ID does not exist" containerID="34bb6dd9415332a7ff06bd21408e100f0b433bfff710c7807d0bbe230e5daf94" Oct 30 13:41:13.366103 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:13.366005 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34bb6dd9415332a7ff06bd21408e100f0b433bfff710c7807d0bbe230e5daf94"} err="failed to get container status \"34bb6dd9415332a7ff06bd21408e100f0b433bfff710c7807d0bbe230e5daf94\": rpc error: code = NotFound desc = could not find container \"34bb6dd9415332a7ff06bd21408e100f0b433bfff710c7807d0bbe230e5daf94\": container with ID starting with 34bb6dd9415332a7ff06bd21408e100f0b433bfff710c7807d0bbe230e5daf94 not found: ID does not exist" Oct 30 13:41:13.431502 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:13.431468 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b"] Oct 30 13:41:13.446814 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:13.446784 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b"] Oct 30 13:41:13.512247 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:13.512216 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e174882a-3e74-4174-99cf-dcf265fb86f6" path="/var/lib/kubelet/pods/e174882a-3e74-4174-99cf-dcf265fb86f6/volumes" Oct 30 13:41:16.835430 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:16.835392 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba8145-p9rz2"] Oct 30 13:41:16.835984 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:16.835962 4931 cpu_manager.go:414] "containerMap: removing container" podUID="440d993b-e62e-4284-955d-2e79adccec2e" containerName="payload-pre-processing" Oct 30 13:41:16.836070 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:16.835985 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="440d993b-e62e-4284-955d-2e79adccec2e" containerName="payload-pre-processing" Oct 30 13:41:16.836070 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:16.836007 4931 cpu_manager.go:414] "containerMap: removing container" podUID="fb8000b8-760a-42bb-9c19-6fdc97354a6a" containerName="revoke-keys" Oct 30 13:41:16.836070 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:16.836016 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb8000b8-760a-42bb-9c19-6fdc97354a6a" containerName="revoke-keys" Oct 30 13:41:16.836070 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:16.836037 4931 cpu_manager.go:414] "containerMap: removing container" podUID="e174882a-3e74-4174-99cf-dcf265fb86f6" containerName="istio-proxy" Oct 30 13:41:16.836070 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:16.836044 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e174882a-3e74-4174-99cf-dcf265fb86f6" containerName="istio-proxy" Oct 30 13:41:16.836367 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:16.836170 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="440d993b-e62e-4284-955d-2e79adccec2e" containerName="payload-pre-processing" Oct 30 13:41:16.836367 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:16.836186 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb8000b8-760a-42bb-9c19-6fdc97354a6a" containerName="revoke-keys" Oct 30 13:41:16.836367 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:16.836198 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="e174882a-3e74-4174-99cf-dcf265fb86f6" containerName="istio-proxy" Oct 30 13:41:16.841129 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:16.841105 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba8145-p9rz2" Oct 30 13:41:16.861474 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:16.861445 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba8145-p9rz2"] Oct 30 13:41:16.871197 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:16.871147 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:41:16.871345 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:16.871252 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:41:16.957323 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:16.957278 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwzf6\" (UniqueName: \"kubernetes.io/projected/89e98db2-186a-4e97-b3ff-e881ef0225da-kube-api-access-fwzf6\") pod \"maas-api-revoke-keys-e2e-ipp-a-ba8145-p9rz2\" (UID: \"89e98db2-186a-4e97-b3ff-e881ef0225da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba8145-p9rz2" Oct 30 13:41:16.957495 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:16.957422 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e98db2-186a-4e97-b3ff-e881ef0225da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ba8145-p9rz2\" (UID: \"89e98db2-186a-4e97-b3ff-e881ef0225da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba8145-p9rz2" Oct 30 13:41:16.957495 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:16.957451 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e98db2-186a-4e97-b3ff-e881ef0225da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ba8145-p9rz2\" (UID: \"89e98db2-186a-4e97-b3ff-e881ef0225da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba8145-p9rz2" Oct 30 13:41:17.058701 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:17.058662 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e98db2-186a-4e97-b3ff-e881ef0225da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ba8145-p9rz2\" (UID: \"89e98db2-186a-4e97-b3ff-e881ef0225da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba8145-p9rz2" Oct 30 13:41:17.058701 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:17.058699 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e98db2-186a-4e97-b3ff-e881ef0225da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ba8145-p9rz2\" (UID: \"89e98db2-186a-4e97-b3ff-e881ef0225da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba8145-p9rz2" Oct 30 13:41:17.058922 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:17.058757 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwzf6\" (UniqueName: \"kubernetes.io/projected/89e98db2-186a-4e97-b3ff-e881ef0225da-kube-api-access-fwzf6\") pod \"maas-api-revoke-keys-e2e-ipp-a-ba8145-p9rz2\" (UID: \"89e98db2-186a-4e97-b3ff-e881ef0225da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba8145-p9rz2" Oct 30 13:41:17.059342 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:17.059320 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e98db2-186a-4e97-b3ff-e881ef0225da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ba8145-p9rz2\" (UID: \"89e98db2-186a-4e97-b3ff-e881ef0225da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba8145-p9rz2" Oct 30 13:41:17.061154 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:17.061130 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e98db2-186a-4e97-b3ff-e881ef0225da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ba8145-p9rz2\" (UID: \"89e98db2-186a-4e97-b3ff-e881ef0225da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba8145-p9rz2" Oct 30 13:41:17.084172 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:17.084144 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwzf6\" (UniqueName: \"kubernetes.io/projected/89e98db2-186a-4e97-b3ff-e881ef0225da-kube-api-access-fwzf6\") pod \"maas-api-revoke-keys-e2e-ipp-a-ba8145-p9rz2\" (UID: \"89e98db2-186a-4e97-b3ff-e881ef0225da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba8145-p9rz2" Oct 30 13:41:17.153711 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:17.153642 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba8145-p9rz2" Oct 30 13:41:17.458957 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:17.458924 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba8145-p9rz2"] Oct 30 13:41:17.460362 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:41:17.460333 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e98db2_186a_4e97_b3ff_e881ef0225da.slice/crio-a8ee7dbc206661605da36718546ad8cc98dc6f8aabc9c4849bad4156afde6c39 WatchSource:0}: Error finding container a8ee7dbc206661605da36718546ad8cc98dc6f8aabc9c4849bad4156afde6c39: Status 404 returned error can't find the container with id a8ee7dbc206661605da36718546ad8cc98dc6f8aabc9c4849bad4156afde6c39 Oct 30 13:41:18.390658 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:18.390623 4931 generic.go:356] "Generic (PLEG): container finished" podID="89e98db2-186a-4e97-b3ff-e881ef0225da" containerID="d978a199f5e9343225a83c1bc228d57230663079f765740623fa79b22f68d92d" exitCode=0 Oct 30 13:41:18.391011 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:18.390699 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba8145-p9rz2" event={"ID":"89e98db2-186a-4e97-b3ff-e881ef0225da","Type":"ContainerDied","Data":"d978a199f5e9343225a83c1bc228d57230663079f765740623fa79b22f68d92d"} Oct 30 13:41:18.391011 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:18.390753 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba8145-p9rz2" event={"ID":"89e98db2-186a-4e97-b3ff-e881ef0225da","Type":"ContainerStarted","Data":"a8ee7dbc206661605da36718546ad8cc98dc6f8aabc9c4849bad4156afde6c39"} Oct 30 13:41:19.531498 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:19.531473 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba8145-p9rz2" Oct 30 13:41:19.685372 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:19.685257 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89e98db2-186a-4e97-b3ff-e881ef0225da-kube-api-access-fwzf6\" (UniqueName: \"kubernetes.io/projected/89e98db2-186a-4e97-b3ff-e881ef0225da-kube-api-access-fwzf6\") pod \"89e98db2-186a-4e97-b3ff-e881ef0225da\" (UID: \"89e98db2-186a-4e97-b3ff-e881ef0225da\") " Oct 30 13:41:19.685551 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:19.685395 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89e98db2-186a-4e97-b3ff-e881ef0225da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e98db2-186a-4e97-b3ff-e881ef0225da-maas-api-openshift-ca\") pod \"89e98db2-186a-4e97-b3ff-e881ef0225da\" (UID: \"89e98db2-186a-4e97-b3ff-e881ef0225da\") " Oct 30 13:41:19.685551 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:19.685420 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89e98db2-186a-4e97-b3ff-e881ef0225da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e98db2-186a-4e97-b3ff-e881ef0225da-maas-api-xks-ca\") pod \"89e98db2-186a-4e97-b3ff-e881ef0225da\" (UID: \"89e98db2-186a-4e97-b3ff-e881ef0225da\") " Oct 30 13:41:19.685801 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:19.685770 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89e98db2-186a-4e97-b3ff-e881ef0225da-maas-api-openshift-ca" pod "89e98db2-186a-4e97-b3ff-e881ef0225da" (UID: "89e98db2-186a-4e97-b3ff-e881ef0225da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:41:19.687506 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:19.687478 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89e98db2-186a-4e97-b3ff-e881ef0225da-kube-api-access-fwzf6" pod "89e98db2-186a-4e97-b3ff-e881ef0225da" (UID: "89e98db2-186a-4e97-b3ff-e881ef0225da"). InnerVolumeSpecName "kube-api-access-fwzf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:19.687618 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:19.687517 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89e98db2-186a-4e97-b3ff-e881ef0225da-maas-api-xks-ca" pod "89e98db2-186a-4e97-b3ff-e881ef0225da" (UID: "89e98db2-186a-4e97-b3ff-e881ef0225da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:41:19.786877 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:19.786841 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwzf6\" (UniqueName: \"kubernetes.io/projected/89e98db2-186a-4e97-b3ff-e881ef0225da-kube-api-access-fwzf6\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:41:19.786877 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:19.786872 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e98db2-186a-4e97-b3ff-e881ef0225da-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:41:19.786877 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:19.786884 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e98db2-186a-4e97-b3ff-e881ef0225da-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:41:20.404370 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:20.404335 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba8145-p9rz2" Oct 30 13:41:20.404558 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:20.404423 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba8145-p9rz2" event={"ID":"89e98db2-186a-4e97-b3ff-e881ef0225da","Type":"ContainerDied","Data":"a8ee7dbc206661605da36718546ad8cc98dc6f8aabc9c4849bad4156afde6c39"} Oct 30 13:41:20.404558 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:20.404508 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8ee7dbc206661605da36718546ad8cc98dc6f8aabc9c4849bad4156afde6c39" Oct 30 13:41:22.203104 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:22.203064 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7a3603-4tpzs"] Oct 30 13:41:22.215554 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:22.215523 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7a3603-4tpzs"] Oct 30 13:41:23.511512 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:23.511479 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5db07c46-5a42-4453-a60d-d3cd9be08b32" path="/var/lib/kubelet/pods/5db07c46-5a42-4453-a60d-d3cd9be08b32/volumes" Oct 30 13:41:30.331870 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:30.331829 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:41:31.130728 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:31.130679 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba8145-p9rz2"] Oct 30 13:41:31.149425 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:31.149388 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba8145-p9rz2"] Oct 30 13:41:31.512734 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:31.512700 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89e98db2-186a-4e97-b3ff-e881ef0225da" path="/var/lib/kubelet/pods/89e98db2-186a-4e97-b3ff-e881ef0225da/volumes" Oct 30 13:41:37.352801 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:41:37.352750 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:41:41.027341 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:41.027299 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:41:44.269962 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:44.269924 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87e742-xl89d"] Oct 30 13:41:44.270402 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:44.270366 4931 cpu_manager.go:414] "containerMap: removing container" podUID="89e98db2-186a-4e97-b3ff-e881ef0225da" containerName="revoke-keys" Oct 30 13:41:44.270402 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:44.270378 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89e98db2-186a-4e97-b3ff-e881ef0225da" containerName="revoke-keys" Oct 30 13:41:44.270479 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:44.270454 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="89e98db2-186a-4e97-b3ff-e881ef0225da" containerName="revoke-keys" Oct 30 13:41:44.273279 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:44.273252 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87e742-xl89d" Oct 30 13:41:44.292760 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:44.292731 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:41:44.306284 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:44.306242 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:41:44.309047 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:44.309023 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87e742-xl89d"] Oct 30 13:41:44.419223 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:44.419190 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b6kn\" (UniqueName: \"kubernetes.io/projected/902d392f-4a4d-4ba1-8a6e-7b53c03b6691-kube-api-access-2b6kn\") pod \"maas-api-revoke-keys-e2e-ipp-b-87e742-xl89d\" (UID: \"902d392f-4a4d-4ba1-8a6e-7b53c03b6691\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87e742-xl89d" Oct 30 13:41:44.419442 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:44.419245 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/902d392f-4a4d-4ba1-8a6e-7b53c03b6691-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-87e742-xl89d\" (UID: \"902d392f-4a4d-4ba1-8a6e-7b53c03b6691\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87e742-xl89d" Oct 30 13:41:44.419442 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:44.419355 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/902d392f-4a4d-4ba1-8a6e-7b53c03b6691-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-87e742-xl89d\" (UID: \"902d392f-4a4d-4ba1-8a6e-7b53c03b6691\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87e742-xl89d" Oct 30 13:41:44.520044 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:44.519954 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/902d392f-4a4d-4ba1-8a6e-7b53c03b6691-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-87e742-xl89d\" (UID: \"902d392f-4a4d-4ba1-8a6e-7b53c03b6691\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87e742-xl89d" Oct 30 13:41:44.520216 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:44.520079 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b6kn\" (UniqueName: \"kubernetes.io/projected/902d392f-4a4d-4ba1-8a6e-7b53c03b6691-kube-api-access-2b6kn\") pod \"maas-api-revoke-keys-e2e-ipp-b-87e742-xl89d\" (UID: \"902d392f-4a4d-4ba1-8a6e-7b53c03b6691\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87e742-xl89d" Oct 30 13:41:44.520216 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:44.520119 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/902d392f-4a4d-4ba1-8a6e-7b53c03b6691-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-87e742-xl89d\" (UID: \"902d392f-4a4d-4ba1-8a6e-7b53c03b6691\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87e742-xl89d" Oct 30 13:41:44.520791 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:44.520764 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/902d392f-4a4d-4ba1-8a6e-7b53c03b6691-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-87e742-xl89d\" (UID: \"902d392f-4a4d-4ba1-8a6e-7b53c03b6691\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87e742-xl89d" Oct 30 13:41:44.522543 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:44.522516 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/902d392f-4a4d-4ba1-8a6e-7b53c03b6691-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-87e742-xl89d\" (UID: \"902d392f-4a4d-4ba1-8a6e-7b53c03b6691\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87e742-xl89d" Oct 30 13:41:44.545012 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:44.544983 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b6kn\" (UniqueName: \"kubernetes.io/projected/902d392f-4a4d-4ba1-8a6e-7b53c03b6691-kube-api-access-2b6kn\") pod \"maas-api-revoke-keys-e2e-ipp-b-87e742-xl89d\" (UID: \"902d392f-4a4d-4ba1-8a6e-7b53c03b6691\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87e742-xl89d" Oct 30 13:41:44.586854 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:44.586826 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87e742-xl89d" Oct 30 13:41:44.889908 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:44.889863 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87e742-xl89d"] Oct 30 13:41:44.891572 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:41:44.891540 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod902d392f_4a4d_4ba1_8a6e_7b53c03b6691.slice/crio-dcdf9bdbfbae8bf2cb03e9f0f61d09420ee3392d440eb96626666a6765752cd1 WatchSource:0}: Error finding container dcdf9bdbfbae8bf2cb03e9f0f61d09420ee3392d440eb96626666a6765752cd1: Status 404 returned error can't find the container with id dcdf9bdbfbae8bf2cb03e9f0f61d09420ee3392d440eb96626666a6765752cd1 Oct 30 13:41:45.573704 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:45.573671 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87e742-xl89d" event={"ID":"902d392f-4a4d-4ba1-8a6e-7b53c03b6691","Type":"ContainerStarted","Data":"b76c6ec384e3beec392a7e76caaf07513a03c72ca99e2aee847ebd4c4073c137"} Oct 30 13:41:45.574136 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:45.573729 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87e742-xl89d" event={"ID":"902d392f-4a4d-4ba1-8a6e-7b53c03b6691","Type":"ContainerStarted","Data":"dcdf9bdbfbae8bf2cb03e9f0f61d09420ee3392d440eb96626666a6765752cd1"} Oct 30 13:41:45.618930 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:45.618872 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87e742-xl89d" podStartSLOduration=1.618857471 podStartE2EDuration="1.618857471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:41:45.615874473 +0000 UTC m=+3404.219372918" watchObservedRunningTime="2026-10-30 13:41:45.618857471 +0000 UTC m=+3404.222355916" Oct 30 13:41:46.582039 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:46.582000 4931 generic.go:356] "Generic (PLEG): container finished" podID="902d392f-4a4d-4ba1-8a6e-7b53c03b6691" containerID="b76c6ec384e3beec392a7e76caaf07513a03c72ca99e2aee847ebd4c4073c137" exitCode=0 Oct 30 13:41:46.582493 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:46.582089 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87e742-xl89d" event={"ID":"902d392f-4a4d-4ba1-8a6e-7b53c03b6691","Type":"ContainerDied","Data":"b76c6ec384e3beec392a7e76caaf07513a03c72ca99e2aee847ebd4c4073c137"} Oct 30 13:41:47.736321 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:47.736293 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87e742-xl89d" Oct 30 13:41:47.852842 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:47.852757 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/902d392f-4a4d-4ba1-8a6e-7b53c03b6691-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/902d392f-4a4d-4ba1-8a6e-7b53c03b6691-maas-api-xks-ca\") pod \"902d392f-4a4d-4ba1-8a6e-7b53c03b6691\" (UID: \"902d392f-4a4d-4ba1-8a6e-7b53c03b6691\") " Oct 30 13:41:47.852842 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:47.852811 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/902d392f-4a4d-4ba1-8a6e-7b53c03b6691-kube-api-access-2b6kn\" (UniqueName: \"kubernetes.io/projected/902d392f-4a4d-4ba1-8a6e-7b53c03b6691-kube-api-access-2b6kn\") pod \"902d392f-4a4d-4ba1-8a6e-7b53c03b6691\" (UID: \"902d392f-4a4d-4ba1-8a6e-7b53c03b6691\") " Oct 30 13:41:47.853041 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:47.852854 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/902d392f-4a4d-4ba1-8a6e-7b53c03b6691-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/902d392f-4a4d-4ba1-8a6e-7b53c03b6691-maas-api-openshift-ca\") pod \"902d392f-4a4d-4ba1-8a6e-7b53c03b6691\" (UID: \"902d392f-4a4d-4ba1-8a6e-7b53c03b6691\") " Oct 30 13:41:47.853327 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:47.853291 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/902d392f-4a4d-4ba1-8a6e-7b53c03b6691-maas-api-openshift-ca" pod "902d392f-4a4d-4ba1-8a6e-7b53c03b6691" (UID: "902d392f-4a4d-4ba1-8a6e-7b53c03b6691"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:41:47.855064 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:47.855039 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/902d392f-4a4d-4ba1-8a6e-7b53c03b6691-kube-api-access-2b6kn" pod "902d392f-4a4d-4ba1-8a6e-7b53c03b6691" (UID: "902d392f-4a4d-4ba1-8a6e-7b53c03b6691"). InnerVolumeSpecName "kube-api-access-2b6kn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:47.855064 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:47.855052 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/902d392f-4a4d-4ba1-8a6e-7b53c03b6691-maas-api-xks-ca" pod "902d392f-4a4d-4ba1-8a6e-7b53c03b6691" (UID: "902d392f-4a4d-4ba1-8a6e-7b53c03b6691"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:41:47.954605 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:47.954565 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b6kn\" (UniqueName: \"kubernetes.io/projected/902d392f-4a4d-4ba1-8a6e-7b53c03b6691-kube-api-access-2b6kn\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:41:47.954605 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:47.954599 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/902d392f-4a4d-4ba1-8a6e-7b53c03b6691-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:41:47.954605 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:47.954609 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/902d392f-4a4d-4ba1-8a6e-7b53c03b6691-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:41:48.595670 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:48.595635 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87e742-xl89d" Oct 30 13:41:48.595836 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:48.595670 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87e742-xl89d" event={"ID":"902d392f-4a4d-4ba1-8a6e-7b53c03b6691","Type":"ContainerDied","Data":"dcdf9bdbfbae8bf2cb03e9f0f61d09420ee3392d440eb96626666a6765752cd1"} Oct 30 13:41:48.595836 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:48.595720 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcdf9bdbfbae8bf2cb03e9f0f61d09420ee3392d440eb96626666a6765752cd1" Oct 30 13:41:56.772825 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:56.772783 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87e742-xl89d"] Oct 30 13:41:56.785230 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:56.785200 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87e742-xl89d"] Oct 30 13:41:57.512573 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:57.512539 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="902d392f-4a4d-4ba1-8a6e-7b53c03b6691" path="/var/lib/kubelet/pods/902d392f-4a4d-4ba1-8a6e-7b53c03b6691/volumes" Oct 30 13:41:59.091826 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:41:59.091787 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:42:07.353433 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:42:07.353390 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:42:19.918322 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:19.918284 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:42:37.353178 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:42:37.353136 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:42:41.996853 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:41.996808 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:42:53.016179 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:53.016100 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8"] Oct 30 13:42:53.038601 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:53.038568 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht"] Oct 30 13:42:53.917306 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:53.917218 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p"] Oct 30 13:42:53.917524 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:53.917498 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p" podUID="3e549a03-6e0a-4068-81f1-8747bc5570a0" containerName="discovery" containerID="cri-o://b3f3073914fd8bd72a3f09fa179292d06caf2a19199b7bb14beb70ec4152b136" gracePeriod=30 Oct 30 13:42:54.205476 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:54.205450 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p" Oct 30 13:42:54.276955 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:54.276919 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e549a03-6e0a-4068-81f1-8747bc5570a0-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3e549a03-6e0a-4068-81f1-8747bc5570a0-istio-kubeconfig\") pod \"3e549a03-6e0a-4068-81f1-8747bc5570a0\" (UID: \"3e549a03-6e0a-4068-81f1-8747bc5570a0\") " Oct 30 13:42:54.277147 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:54.277042 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e549a03-6e0a-4068-81f1-8747bc5570a0-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e549a03-6e0a-4068-81f1-8747bc5570a0-local-certs\") pod \"3e549a03-6e0a-4068-81f1-8747bc5570a0\" (UID: \"3e549a03-6e0a-4068-81f1-8747bc5570a0\") " Oct 30 13:42:54.277147 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:54.277084 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e549a03-6e0a-4068-81f1-8747bc5570a0-kube-api-access-nd2m4\" (UniqueName: \"kubernetes.io/projected/3e549a03-6e0a-4068-81f1-8747bc5570a0-kube-api-access-nd2m4\") pod \"3e549a03-6e0a-4068-81f1-8747bc5570a0\" (UID: \"3e549a03-6e0a-4068-81f1-8747bc5570a0\") " Oct 30 13:42:54.277147 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:54.277111 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e549a03-6e0a-4068-81f1-8747bc5570a0-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3e549a03-6e0a-4068-81f1-8747bc5570a0-istio-csr-dns-cert\") pod \"3e549a03-6e0a-4068-81f1-8747bc5570a0\" (UID: \"3e549a03-6e0a-4068-81f1-8747bc5570a0\") " Oct 30 13:42:54.277147 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:54.277140 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e549a03-6e0a-4068-81f1-8747bc5570a0-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3e549a03-6e0a-4068-81f1-8747bc5570a0-istio-csr-ca-configmap\") pod \"3e549a03-6e0a-4068-81f1-8747bc5570a0\" (UID: \"3e549a03-6e0a-4068-81f1-8747bc5570a0\") " Oct 30 13:42:54.277428 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:54.277175 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e549a03-6e0a-4068-81f1-8747bc5570a0-istio-token\" (UniqueName: \"kubernetes.io/projected/3e549a03-6e0a-4068-81f1-8747bc5570a0-istio-token\") pod \"3e549a03-6e0a-4068-81f1-8747bc5570a0\" (UID: \"3e549a03-6e0a-4068-81f1-8747bc5570a0\") " Oct 30 13:42:54.277428 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:54.277260 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e549a03-6e0a-4068-81f1-8747bc5570a0-cacerts\" (UniqueName: \"kubernetes.io/secret/3e549a03-6e0a-4068-81f1-8747bc5570a0-cacerts\") pod \"3e549a03-6e0a-4068-81f1-8747bc5570a0\" (UID: \"3e549a03-6e0a-4068-81f1-8747bc5570a0\") " Oct 30 13:42:54.277576 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:54.277549 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e549a03-6e0a-4068-81f1-8747bc5570a0-istio-csr-ca-configmap" pod "3e549a03-6e0a-4068-81f1-8747bc5570a0" (UID: "3e549a03-6e0a-4068-81f1-8747bc5570a0"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:54.277742 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:54.277713 4931 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3e549a03-6e0a-4068-81f1-8747bc5570a0-istio-csr-ca-configmap\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:42:54.279749 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:54.279708 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e549a03-6e0a-4068-81f1-8747bc5570a0-kube-api-access-nd2m4" pod "3e549a03-6e0a-4068-81f1-8747bc5570a0" (UID: "3e549a03-6e0a-4068-81f1-8747bc5570a0"). InnerVolumeSpecName "kube-api-access-nd2m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:54.280104 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:54.280074 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e549a03-6e0a-4068-81f1-8747bc5570a0-local-certs" pod "3e549a03-6e0a-4068-81f1-8747bc5570a0" (UID: "3e549a03-6e0a-4068-81f1-8747bc5570a0"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:54.280215 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:54.280118 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e549a03-6e0a-4068-81f1-8747bc5570a0-cacerts" pod "3e549a03-6e0a-4068-81f1-8747bc5570a0" (UID: "3e549a03-6e0a-4068-81f1-8747bc5570a0"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:42:54.280215 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:54.280163 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e549a03-6e0a-4068-81f1-8747bc5570a0-istio-csr-dns-cert" pod "3e549a03-6e0a-4068-81f1-8747bc5570a0" (UID: "3e549a03-6e0a-4068-81f1-8747bc5570a0"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:42:54.280347 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:54.280289 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e549a03-6e0a-4068-81f1-8747bc5570a0-istio-token" pod "3e549a03-6e0a-4068-81f1-8747bc5570a0" (UID: "3e549a03-6e0a-4068-81f1-8747bc5570a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:54.284385 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:54.284361 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e549a03-6e0a-4068-81f1-8747bc5570a0-istio-kubeconfig" pod "3e549a03-6e0a-4068-81f1-8747bc5570a0" (UID: "3e549a03-6e0a-4068-81f1-8747bc5570a0"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:42:54.378533 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:54.378456 4931 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e549a03-6e0a-4068-81f1-8747bc5570a0-local-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:42:54.378533 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:54.378485 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd2m4\" (UniqueName: \"kubernetes.io/projected/3e549a03-6e0a-4068-81f1-8747bc5570a0-kube-api-access-nd2m4\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:42:54.378533 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:54.378495 4931 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3e549a03-6e0a-4068-81f1-8747bc5570a0-istio-csr-dns-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:42:54.378533 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:54.378504 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e549a03-6e0a-4068-81f1-8747bc5570a0-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:42:54.378533 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:54.378513 4931 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3e549a03-6e0a-4068-81f1-8747bc5570a0-cacerts\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:42:54.378533 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:54.378520 4931 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3e549a03-6e0a-4068-81f1-8747bc5570a0-istio-kubeconfig\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:42:55.042118 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:55.042083 4931 generic.go:356] "Generic (PLEG): container finished" podID="3e549a03-6e0a-4068-81f1-8747bc5570a0" containerID="b3f3073914fd8bd72a3f09fa179292d06caf2a19199b7bb14beb70ec4152b136" exitCode=0 Oct 30 13:42:55.042341 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:55.042142 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p" Oct 30 13:42:55.042341 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:55.042167 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p" event={"ID":"3e549a03-6e0a-4068-81f1-8747bc5570a0","Type":"ContainerDied","Data":"b3f3073914fd8bd72a3f09fa179292d06caf2a19199b7bb14beb70ec4152b136"} Oct 30 13:42:55.042341 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:55.042223 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p" event={"ID":"3e549a03-6e0a-4068-81f1-8747bc5570a0","Type":"ContainerDied","Data":"96160c80f36dc9f0fd96b58ef4bdb995e23af732c079612fe7b12366457ea5ea"} Oct 30 13:42:55.042341 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:55.042252 4931 scope.go:122] "RemoveContainer" containerID="b3f3073914fd8bd72a3f09fa179292d06caf2a19199b7bb14beb70ec4152b136" Oct 30 13:42:55.053947 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:55.053926 4931 scope.go:122] "RemoveContainer" containerID="b3f3073914fd8bd72a3f09fa179292d06caf2a19199b7bb14beb70ec4152b136" Oct 30 13:42:55.054242 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:42:55.054223 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3f3073914fd8bd72a3f09fa179292d06caf2a19199b7bb14beb70ec4152b136\": container with ID starting with b3f3073914fd8bd72a3f09fa179292d06caf2a19199b7bb14beb70ec4152b136 not found: ID does not exist" containerID="b3f3073914fd8bd72a3f09fa179292d06caf2a19199b7bb14beb70ec4152b136" Oct 30 13:42:55.054350 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:55.054249 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3f3073914fd8bd72a3f09fa179292d06caf2a19199b7bb14beb70ec4152b136"} err="failed to get container status \"b3f3073914fd8bd72a3f09fa179292d06caf2a19199b7bb14beb70ec4152b136\": rpc error: code = NotFound desc = could not find container \"b3f3073914fd8bd72a3f09fa179292d06caf2a19199b7bb14beb70ec4152b136\": container with ID starting with b3f3073914fd8bd72a3f09fa179292d06caf2a19199b7bb14beb70ec4152b136 not found: ID does not exist" Oct 30 13:42:55.114146 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:55.114114 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p"] Oct 30 13:42:55.127237 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:55.127208 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p"] Oct 30 13:42:55.513448 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:55.513414 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e549a03-6e0a-4068-81f1-8747bc5570a0" path="/var/lib/kubelet/pods/3e549a03-6e0a-4068-81f1-8747bc5570a0/volumes" Oct 30 13:42:58.648203 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:42:58.648167 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78"] Oct 30 13:43:03.648585 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:03.648541 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78" podUID="c3d6937c-4924-4bb7-9449-6c4df4306904" containerName="payload-processing" containerID="cri-o://8fe3e9ad466c7c28d914b2c28979c8cda7b4b7dacac69fc4070474b540ae0f0e" gracePeriod=25 Oct 30 13:43:04.372976 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:04.372938 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78" podUID="c3d6937c-4924-4bb7-9449-6c4df4306904" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:43:07.351249 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:43:07.351213 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:43:08.036874 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.036822 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" podUID="5a504237-e5a3-491f-8f84-0419cf502f9a" containerName="main" containerID="cri-o://87de98e52fbad25f10a8eed84ca4beb9ffad207d4a2d5745c82dd84cc5d416a6" gracePeriod=45 Oct 30 13:43:08.059350 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.059319 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" podUID="b1e86eea-4824-4c50-8ad3-2a15c3365c3d" containerName="main" containerID="cri-o://9c10004e68dad22efe8c38e4ffcda9a991b30f1d514b10df9e39fa445e51a593" gracePeriod=45 Oct 30 13:43:08.163044 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.163009 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" podUID="b1e86eea-4824-4c50-8ad3-2a15c3365c3d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": dial tcp 10.132.0.56:8000: connect: connection refused" Oct 30 13:43:08.217659 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.217611 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" podUID="5a504237-e5a3-491f-8f84-0419cf502f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": dial tcp 10.132.0.73:8000: connect: connection refused" Oct 30 13:43:08.314128 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.314103 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:43:08.332830 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.332800 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" Oct 30 13:43:08.413635 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.413602 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-dshm\") pod \"b1e86eea-4824-4c50-8ad3-2a15c3365c3d\" (UID: \"b1e86eea-4824-4c50-8ad3-2a15c3365c3d\") " Oct 30 13:43:08.414190 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.413649 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a504237-e5a3-491f-8f84-0419cf502f9a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a504237-e5a3-491f-8f84-0419cf502f9a-tmp-dir\") pod \"5a504237-e5a3-491f-8f84-0419cf502f9a\" (UID: \"5a504237-e5a3-491f-8f84-0419cf502f9a\") " Oct 30 13:43:08.414190 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.413682 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a504237-e5a3-491f-8f84-0419cf502f9a-home\" (UniqueName: \"kubernetes.io/empty-dir/5a504237-e5a3-491f-8f84-0419cf502f9a-home\") pod \"5a504237-e5a3-491f-8f84-0419cf502f9a\" (UID: \"5a504237-e5a3-491f-8f84-0419cf502f9a\") " Oct 30 13:43:08.414190 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.413723 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-tmp-dir\") pod \"b1e86eea-4824-4c50-8ad3-2a15c3365c3d\" (UID: \"b1e86eea-4824-4c50-8ad3-2a15c3365c3d\") " Oct 30 13:43:08.414190 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.413750 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a504237-e5a3-491f-8f84-0419cf502f9a-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a504237-e5a3-491f-8f84-0419cf502f9a-tls-certs\") pod \"5a504237-e5a3-491f-8f84-0419cf502f9a\" (UID: \"5a504237-e5a3-491f-8f84-0419cf502f9a\") " Oct 30 13:43:08.414190 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.413783 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-home\" (UniqueName: \"kubernetes.io/empty-dir/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-home\") pod \"b1e86eea-4824-4c50-8ad3-2a15c3365c3d\" (UID: \"b1e86eea-4824-4c50-8ad3-2a15c3365c3d\") " Oct 30 13:43:08.414190 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.413825 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-kube-api-access-8glx9\" (UniqueName: \"kubernetes.io/projected/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-kube-api-access-8glx9\") pod \"b1e86eea-4824-4c50-8ad3-2a15c3365c3d\" (UID: \"b1e86eea-4824-4c50-8ad3-2a15c3365c3d\") " Oct 30 13:43:08.414190 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.413852 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a504237-e5a3-491f-8f84-0419cf502f9a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a504237-e5a3-491f-8f84-0419cf502f9a-dshm\") pod \"5a504237-e5a3-491f-8f84-0419cf502f9a\" (UID: \"5a504237-e5a3-491f-8f84-0419cf502f9a\") " Oct 30 13:43:08.414190 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.413913 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a504237-e5a3-491f-8f84-0419cf502f9a-kube-api-access-tsd6z\" (UniqueName: \"kubernetes.io/projected/5a504237-e5a3-491f-8f84-0419cf502f9a-kube-api-access-tsd6z\") pod \"5a504237-e5a3-491f-8f84-0419cf502f9a\" (UID: \"5a504237-e5a3-491f-8f84-0419cf502f9a\") " Oct 30 13:43:08.414190 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.413943 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-tls-certs\" (UniqueName: \"kubernetes.io/secret/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-tls-certs\") pod \"b1e86eea-4824-4c50-8ad3-2a15c3365c3d\" (UID: \"b1e86eea-4824-4c50-8ad3-2a15c3365c3d\") " Oct 30 13:43:08.414190 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.413992 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a504237-e5a3-491f-8f84-0419cf502f9a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a504237-e5a3-491f-8f84-0419cf502f9a-model-cache\") pod \"5a504237-e5a3-491f-8f84-0419cf502f9a\" (UID: \"5a504237-e5a3-491f-8f84-0419cf502f9a\") " Oct 30 13:43:08.414190 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.414032 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-model-cache\") pod \"b1e86eea-4824-4c50-8ad3-2a15c3365c3d\" (UID: \"b1e86eea-4824-4c50-8ad3-2a15c3365c3d\") " Oct 30 13:43:08.415577 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.414491 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-model-cache" pod "b1e86eea-4824-4c50-8ad3-2a15c3365c3d" (UID: "b1e86eea-4824-4c50-8ad3-2a15c3365c3d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:08.415577 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.414633 4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-model-cache\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:43:08.415577 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.414830 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a504237-e5a3-491f-8f84-0419cf502f9a-tmp-dir" pod "5a504237-e5a3-491f-8f84-0419cf502f9a" (UID: "5a504237-e5a3-491f-8f84-0419cf502f9a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:08.415886 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.415863 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a504237-e5a3-491f-8f84-0419cf502f9a-home" pod "5a504237-e5a3-491f-8f84-0419cf502f9a" (UID: "5a504237-e5a3-491f-8f84-0419cf502f9a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:08.416085 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.416056 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-tmp-dir" pod "b1e86eea-4824-4c50-8ad3-2a15c3365c3d" (UID: "b1e86eea-4824-4c50-8ad3-2a15c3365c3d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:08.419303 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.416615 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a504237-e5a3-491f-8f84-0419cf502f9a-model-cache" pod "5a504237-e5a3-491f-8f84-0419cf502f9a" (UID: "5a504237-e5a3-491f-8f84-0419cf502f9a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:08.419303 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.416628 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-home" pod "b1e86eea-4824-4c50-8ad3-2a15c3365c3d" (UID: "b1e86eea-4824-4c50-8ad3-2a15c3365c3d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:08.419303 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.419038 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-tls-certs" pod "b1e86eea-4824-4c50-8ad3-2a15c3365c3d" (UID: "b1e86eea-4824-4c50-8ad3-2a15c3365c3d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:43:08.419303 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.419103 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a504237-e5a3-491f-8f84-0419cf502f9a-dshm" pod "5a504237-e5a3-491f-8f84-0419cf502f9a" (UID: "5a504237-e5a3-491f-8f84-0419cf502f9a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:08.419303 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.419151 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-kube-api-access-8glx9" pod "b1e86eea-4824-4c50-8ad3-2a15c3365c3d" (UID: "b1e86eea-4824-4c50-8ad3-2a15c3365c3d"). InnerVolumeSpecName "kube-api-access-8glx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:08.419632 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.419434 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a504237-e5a3-491f-8f84-0419cf502f9a-kube-api-access-tsd6z" pod "5a504237-e5a3-491f-8f84-0419cf502f9a" (UID: "5a504237-e5a3-491f-8f84-0419cf502f9a"). InnerVolumeSpecName "kube-api-access-tsd6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:08.421390 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.421366 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-dshm" pod "b1e86eea-4824-4c50-8ad3-2a15c3365c3d" (UID: "b1e86eea-4824-4c50-8ad3-2a15c3365c3d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:08.421482 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.421426 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a504237-e5a3-491f-8f84-0419cf502f9a-tls-certs" pod "5a504237-e5a3-491f-8f84-0419cf502f9a" (UID: "5a504237-e5a3-491f-8f84-0419cf502f9a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:43:08.515530 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.515497 4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a504237-e5a3-491f-8f84-0419cf502f9a-tmp-dir\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:43:08.515530 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.515527 4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a504237-e5a3-491f-8f84-0419cf502f9a-home\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:43:08.515530 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.515537 4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-tmp-dir\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:43:08.515787 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.515545 4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a504237-e5a3-491f-8f84-0419cf502f9a-tls-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:43:08.515787 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.515553 4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-home\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:43:08.515787 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.515561 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8glx9\" (UniqueName: \"kubernetes.io/projected/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-kube-api-access-8glx9\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:43:08.515787 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.515569 4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a504237-e5a3-491f-8f84-0419cf502f9a-dshm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:43:08.515787 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.515577 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsd6z\" (UniqueName: \"kubernetes.io/projected/5a504237-e5a3-491f-8f84-0419cf502f9a-kube-api-access-tsd6z\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:43:08.515787 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.515587 4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-tls-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:43:08.515787 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.515595 4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a504237-e5a3-491f-8f84-0419cf502f9a-model-cache\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:43:08.515787 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:08.515602 4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1e86eea-4824-4c50-8ad3-2a15c3365c3d-dshm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:43:09.127873 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:09.127839 4931 generic.go:356] "Generic (PLEG): container finished" podID="b1e86eea-4824-4c50-8ad3-2a15c3365c3d" containerID="9c10004e68dad22efe8c38e4ffcda9a991b30f1d514b10df9e39fa445e51a593" exitCode=0 Oct 30 13:43:09.128060 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:09.127884 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" Oct 30 13:43:09.128060 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:09.127934 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" event={"ID":"b1e86eea-4824-4c50-8ad3-2a15c3365c3d","Type":"ContainerDied","Data":"9c10004e68dad22efe8c38e4ffcda9a991b30f1d514b10df9e39fa445e51a593"} Oct 30 13:43:09.128060 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:09.127982 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht" event={"ID":"b1e86eea-4824-4c50-8ad3-2a15c3365c3d","Type":"ContainerDied","Data":"f3e6c8e59a8d06b63e09cca8e3a5e72a7b772e8c6edc51e5c1f21adaaa46213a"} Oct 30 13:43:09.128060 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:09.128004 4931 scope.go:122] "RemoveContainer" containerID="9c10004e68dad22efe8c38e4ffcda9a991b30f1d514b10df9e39fa445e51a593" Oct 30 13:43:09.129442 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:09.129419 4931 generic.go:356] "Generic (PLEG): container finished" podID="5a504237-e5a3-491f-8f84-0419cf502f9a" containerID="87de98e52fbad25f10a8eed84ca4beb9ffad207d4a2d5745c82dd84cc5d416a6" exitCode=0 Oct 30 13:43:09.129566 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:09.129448 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" event={"ID":"5a504237-e5a3-491f-8f84-0419cf502f9a","Type":"ContainerDied","Data":"87de98e52fbad25f10a8eed84ca4beb9ffad207d4a2d5745c82dd84cc5d416a6"} Oct 30 13:43:09.129566 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:09.129475 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" Oct 30 13:43:09.129566 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:09.129489 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8" event={"ID":"5a504237-e5a3-491f-8f84-0419cf502f9a","Type":"ContainerDied","Data":"1930ef76186fc36438946a99dcf84ebe19e8f3cfef0782b33617bba1819a2be3"} Oct 30 13:43:09.138929 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:09.138913 4931 scope.go:122] "RemoveContainer" containerID="9c10004e68dad22efe8c38e4ffcda9a991b30f1d514b10df9e39fa445e51a593" Oct 30 13:43:09.139172 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:43:09.139149 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c10004e68dad22efe8c38e4ffcda9a991b30f1d514b10df9e39fa445e51a593\": container with ID starting with 9c10004e68dad22efe8c38e4ffcda9a991b30f1d514b10df9e39fa445e51a593 not found: ID does not exist" containerID="9c10004e68dad22efe8c38e4ffcda9a991b30f1d514b10df9e39fa445e51a593" Oct 30 13:43:09.139245 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:09.139186 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c10004e68dad22efe8c38e4ffcda9a991b30f1d514b10df9e39fa445e51a593"} err="failed to get container status \"9c10004e68dad22efe8c38e4ffcda9a991b30f1d514b10df9e39fa445e51a593\": rpc error: code = NotFound desc = could not find container \"9c10004e68dad22efe8c38e4ffcda9a991b30f1d514b10df9e39fa445e51a593\": container with ID starting with 9c10004e68dad22efe8c38e4ffcda9a991b30f1d514b10df9e39fa445e51a593 not found: ID does not exist" Oct 30 13:43:09.139245 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:09.139211 4931 scope.go:122] "RemoveContainer" containerID="87de98e52fbad25f10a8eed84ca4beb9ffad207d4a2d5745c82dd84cc5d416a6" Oct 30 13:43:09.148889 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:09.148873 4931 scope.go:122] "RemoveContainer" containerID="87de98e52fbad25f10a8eed84ca4beb9ffad207d4a2d5745c82dd84cc5d416a6" Oct 30 13:43:09.149133 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:43:09.149115 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87de98e52fbad25f10a8eed84ca4beb9ffad207d4a2d5745c82dd84cc5d416a6\": container with ID starting with 87de98e52fbad25f10a8eed84ca4beb9ffad207d4a2d5745c82dd84cc5d416a6 not found: ID does not exist" containerID="87de98e52fbad25f10a8eed84ca4beb9ffad207d4a2d5745c82dd84cc5d416a6" Oct 30 13:43:09.149192 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:09.149138 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87de98e52fbad25f10a8eed84ca4beb9ffad207d4a2d5745c82dd84cc5d416a6"} err="failed to get container status \"87de98e52fbad25f10a8eed84ca4beb9ffad207d4a2d5745c82dd84cc5d416a6\": rpc error: code = NotFound desc = could not find container \"87de98e52fbad25f10a8eed84ca4beb9ffad207d4a2d5745c82dd84cc5d416a6\": container with ID starting with 87de98e52fbad25f10a8eed84ca4beb9ffad207d4a2d5745c82dd84cc5d416a6 not found: ID does not exist" Oct 30 13:43:09.189365 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:09.189339 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8"] Oct 30 13:43:09.202352 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:09.202331 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-67f56e/distinct-w4-67f56e-kserve-6487fd96db-cl7z8"] Oct 30 13:43:09.238832 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:09.238810 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht"] Oct 30 13:43:09.249320 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:09.244393 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-67f56e/simulator-w4-67f56e-kserve-69f8c59df8-tt5ht"] Oct 30 13:43:09.513398 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:09.513363 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a504237-e5a3-491f-8f84-0419cf502f9a" path="/var/lib/kubelet/pods/5a504237-e5a3-491f-8f84-0419cf502f9a/volumes" Oct 30 13:43:09.514027 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:09.514003 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1e86eea-4824-4c50-8ad3-2a15c3365c3d" path="/var/lib/kubelet/pods/b1e86eea-4824-4c50-8ad3-2a15c3365c3d/volumes" Oct 30 13:43:12.595908 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:12.595873 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:43:14.373226 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:14.373185 4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78" podUID="c3d6937c-4924-4bb7-9449-6c4df4306904" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:43:14.958127 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:14.958088 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx"] Oct 30 13:43:14.958369 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:14.958329 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx" podUID="996e2b88-ae9e-4e95-8a65-4180de0588b0" containerName="maas-api" containerID="cri-o://f77d9c53b26327d3144a16df810b5d0726b031c2da39f49f5bf081f789407650" gracePeriod=30 Oct 30 13:43:15.179424 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:15.179388 4931 generic.go:356] "Generic (PLEG): container finished" podID="996e2b88-ae9e-4e95-8a65-4180de0588b0" containerID="f77d9c53b26327d3144a16df810b5d0726b031c2da39f49f5bf081f789407650" exitCode=0 Oct 30 13:43:15.179599 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:15.179457 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx" event={"ID":"996e2b88-ae9e-4e95-8a65-4180de0588b0","Type":"ContainerDied","Data":"f77d9c53b26327d3144a16df810b5d0726b031c2da39f49f5bf081f789407650"} Oct 30 13:43:15.216370 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:15.216299 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx" Oct 30 13:43:15.276580 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:15.276550 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/996e2b88-ae9e-4e95-8a65-4180de0588b0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/996e2b88-ae9e-4e95-8a65-4180de0588b0-maas-api-tls\") pod \"996e2b88-ae9e-4e95-8a65-4180de0588b0\" (UID: \"996e2b88-ae9e-4e95-8a65-4180de0588b0\") " Oct 30 13:43:15.276759 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:15.276601 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/996e2b88-ae9e-4e95-8a65-4180de0588b0-kube-api-access-t6c4d\" (UniqueName: \"kubernetes.io/projected/996e2b88-ae9e-4e95-8a65-4180de0588b0-kube-api-access-t6c4d\") pod \"996e2b88-ae9e-4e95-8a65-4180de0588b0\" (UID: \"996e2b88-ae9e-4e95-8a65-4180de0588b0\") " Oct 30 13:43:15.276759 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:15.276649 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/996e2b88-ae9e-4e95-8a65-4180de0588b0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/996e2b88-ae9e-4e95-8a65-4180de0588b0-openshift-service-ca\") pod \"996e2b88-ae9e-4e95-8a65-4180de0588b0\" (UID: \"996e2b88-ae9e-4e95-8a65-4180de0588b0\") " Oct 30 13:43:15.276759 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:15.276701 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/996e2b88-ae9e-4e95-8a65-4180de0588b0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/996e2b88-ae9e-4e95-8a65-4180de0588b0-metrics-certs\") pod \"996e2b88-ae9e-4e95-8a65-4180de0588b0\" (UID: \"996e2b88-ae9e-4e95-8a65-4180de0588b0\") " Oct 30 13:43:15.277081 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:15.277044 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/996e2b88-ae9e-4e95-8a65-4180de0588b0-openshift-service-ca" pod "996e2b88-ae9e-4e95-8a65-4180de0588b0" (UID: "996e2b88-ae9e-4e95-8a65-4180de0588b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:43:15.279022 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:15.278997 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/996e2b88-ae9e-4e95-8a65-4180de0588b0-maas-api-tls" pod "996e2b88-ae9e-4e95-8a65-4180de0588b0" (UID: "996e2b88-ae9e-4e95-8a65-4180de0588b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:43:15.279022 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:15.279008 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/996e2b88-ae9e-4e95-8a65-4180de0588b0-kube-api-access-t6c4d" pod "996e2b88-ae9e-4e95-8a65-4180de0588b0" (UID: "996e2b88-ae9e-4e95-8a65-4180de0588b0"). InnerVolumeSpecName "kube-api-access-t6c4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:15.279184 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:15.279014 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/996e2b88-ae9e-4e95-8a65-4180de0588b0-metrics-certs" pod "996e2b88-ae9e-4e95-8a65-4180de0588b0" (UID: "996e2b88-ae9e-4e95-8a65-4180de0588b0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:43:15.377713 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:15.377675 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/996e2b88-ae9e-4e95-8a65-4180de0588b0-maas-api-tls\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:43:15.377713 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:15.377709 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6c4d\" (UniqueName: \"kubernetes.io/projected/996e2b88-ae9e-4e95-8a65-4180de0588b0-kube-api-access-t6c4d\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:43:15.377713 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:15.377719 4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/996e2b88-ae9e-4e95-8a65-4180de0588b0-openshift-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:43:15.378439 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:15.377728 4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/996e2b88-ae9e-4e95-8a65-4180de0588b0-metrics-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:43:16.187372 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:16.187337 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx" Oct 30 13:43:16.187372 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:16.187353 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx" event={"ID":"996e2b88-ae9e-4e95-8a65-4180de0588b0","Type":"ContainerDied","Data":"9a7b63b825fe8511a7fa60b3cd9b61143bda3fdeb3a4b89319a3d09f6e5a5082"} Oct 30 13:43:16.187668 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:16.187426 4931 scope.go:122] "RemoveContainer" containerID="f77d9c53b26327d3144a16df810b5d0726b031c2da39f49f5bf081f789407650" Oct 30 13:43:16.249404 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:16.249371 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx"] Oct 30 13:43:16.262799 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:16.262767 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-67f56e-5cd9c9f8d7-kqhmx"] Oct 30 13:43:17.511766 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:17.511719 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="996e2b88-ae9e-4e95-8a65-4180de0588b0" path="/var/lib/kubelet/pods/996e2b88-ae9e-4e95-8a65-4180de0588b0/volumes" Oct 30 13:43:21.221513 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:21.221480 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:43:23.898358 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:23.898332 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78" Oct 30 13:43:24.052785 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:24.052747 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3d6937c-4924-4bb7-9449-6c4df4306904-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3d6937c-4924-4bb7-9449-6c4df4306904-plugins-config-volume\") pod \"c3d6937c-4924-4bb7-9449-6c4df4306904\" (UID: \"c3d6937c-4924-4bb7-9449-6c4df4306904\") " Oct 30 13:43:24.052785 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:24.052792 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3d6937c-4924-4bb7-9449-6c4df4306904-kube-api-access-n5kl4\" (UniqueName: \"kubernetes.io/projected/c3d6937c-4924-4bb7-9449-6c4df4306904-kube-api-access-n5kl4\") pod \"c3d6937c-4924-4bb7-9449-6c4df4306904\" (UID: \"c3d6937c-4924-4bb7-9449-6c4df4306904\") " Oct 30 13:43:24.053127 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:24.053100 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3d6937c-4924-4bb7-9449-6c4df4306904-plugins-config-volume" pod "c3d6937c-4924-4bb7-9449-6c4df4306904" (UID: "c3d6937c-4924-4bb7-9449-6c4df4306904"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:43:24.055012 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:24.054988 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3d6937c-4924-4bb7-9449-6c4df4306904-kube-api-access-n5kl4" pod "c3d6937c-4924-4bb7-9449-6c4df4306904" (UID: "c3d6937c-4924-4bb7-9449-6c4df4306904"). InnerVolumeSpecName "kube-api-access-n5kl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:24.154553 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:24.154521 4931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3d6937c-4924-4bb7-9449-6c4df4306904-plugins-config-volume\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:43:24.154553 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:24.154550 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5kl4\" (UniqueName: \"kubernetes.io/projected/c3d6937c-4924-4bb7-9449-6c4df4306904-kube-api-access-n5kl4\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:43:24.242922 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:24.242888 4931 generic.go:356] "Generic (PLEG): container finished" podID="c3d6937c-4924-4bb7-9449-6c4df4306904" containerID="8fe3e9ad466c7c28d914b2c28979c8cda7b4b7dacac69fc4070474b540ae0f0e" exitCode=0 Oct 30 13:43:24.243104 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:24.242937 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78" Oct 30 13:43:24.243104 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:24.242979 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78" event={"ID":"c3d6937c-4924-4bb7-9449-6c4df4306904","Type":"ContainerDied","Data":"8fe3e9ad466c7c28d914b2c28979c8cda7b4b7dacac69fc4070474b540ae0f0e"} Oct 30 13:43:24.243104 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:24.243038 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78" event={"ID":"c3d6937c-4924-4bb7-9449-6c4df4306904","Type":"ContainerDied","Data":"9ed968ef83be3bec0f5acab3d78021b94fcb33addc3b962bc7fbbc35ce1ba5ac"} Oct 30 13:43:24.243104 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:24.243067 4931 scope.go:122] "RemoveContainer" containerID="8fe3e9ad466c7c28d914b2c28979c8cda7b4b7dacac69fc4070474b540ae0f0e" Oct 30 13:43:24.253747 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:24.253728 4931 scope.go:122] "RemoveContainer" containerID="8fe3e9ad466c7c28d914b2c28979c8cda7b4b7dacac69fc4070474b540ae0f0e" Oct 30 13:43:24.254031 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:43:24.254009 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fe3e9ad466c7c28d914b2c28979c8cda7b4b7dacac69fc4070474b540ae0f0e\": container with ID starting with 8fe3e9ad466c7c28d914b2c28979c8cda7b4b7dacac69fc4070474b540ae0f0e not found: ID does not exist" containerID="8fe3e9ad466c7c28d914b2c28979c8cda7b4b7dacac69fc4070474b540ae0f0e" Oct 30 13:43:24.254116 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:24.254041 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fe3e9ad466c7c28d914b2c28979c8cda7b4b7dacac69fc4070474b540ae0f0e"} err="failed to get container status \"8fe3e9ad466c7c28d914b2c28979c8cda7b4b7dacac69fc4070474b540ae0f0e\": rpc error: code = NotFound desc = could not find container \"8fe3e9ad466c7c28d914b2c28979c8cda7b4b7dacac69fc4070474b540ae0f0e\": container with ID starting with 8fe3e9ad466c7c28d914b2c28979c8cda7b4b7dacac69fc4070474b540ae0f0e not found: ID does not exist" Oct 30 13:43:24.302065 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:24.302035 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78"] Oct 30 13:43:24.307026 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:24.306965 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78"] Oct 30 13:43:25.512283 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:25.512236 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3d6937c-4924-4bb7-9449-6c4df4306904" path="/var/lib/kubelet/pods/c3d6937c-4924-4bb7-9449-6c4df4306904/volumes" Oct 30 13:43:37.351645 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:43:37.351604 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:43:38.033820 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:38.033787 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d1153b2-qqh44"] Oct 30 13:43:38.047077 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:38.047047 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d1153b2-qqh44"] Oct 30 13:43:39.043749 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:39.043716 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcd6f9cc-dwvsn"] Oct 30 13:43:39.056633 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:39.056601 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcd6f9cc-dwvsn"] Oct 30 13:43:39.518888 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:39.518849 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20ca3800-3b1d-46de-a14c-4b9e2da3cd47" path="/var/lib/kubelet/pods/20ca3800-3b1d-46de-a14c-4b9e2da3cd47/volumes" Oct 30 13:43:39.519311 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:43:39.519293 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2bc558d-aaa3-4f4d-9834-b4300517b1af" path="/var/lib/kubelet/pods/b2bc558d-aaa3-4f4d-9834-b4300517b1af/volumes" Oct 30 13:44:04.714466 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:44:04.714431 4931 scope.go:122] "RemoveContainer" containerID="6bdcc7a3fc7ce304c22f418b0d76cef2f467f163526c873d3e5b19f2a5639a82" Oct 30 13:44:07.351560 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:44:07.351519 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:44:34.828637 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:44:34.828605 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:44:37.350970 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:44:37.350931 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:44:52.996167 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:44:52.996122 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:44:57.317145 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:44:57.317107 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:45:01.985250 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:01.985214 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2tbzn"] Oct 30 13:45:01.985786 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:01.985601 4931 cpu_manager.go:414] "containerMap: removing container" podUID="c3d6937c-4924-4bb7-9449-6c4df4306904" containerName="payload-processing" Oct 30 13:45:01.985786 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:01.985621 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3d6937c-4924-4bb7-9449-6c4df4306904" containerName="payload-processing" Oct 30 13:45:01.985786 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:01.985638 4931 cpu_manager.go:414] "containerMap: removing container" podUID="5a504237-e5a3-491f-8f84-0419cf502f9a" containerName="main" Oct 30 13:45:01.985786 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:01.985644 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a504237-e5a3-491f-8f84-0419cf502f9a" containerName="main" Oct 30 13:45:01.985786 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:01.985656 4931 cpu_manager.go:414] "containerMap: removing container" podUID="996e2b88-ae9e-4e95-8a65-4180de0588b0" containerName="maas-api" Oct 30 13:45:01.985786 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:01.985661 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="996e2b88-ae9e-4e95-8a65-4180de0588b0" containerName="maas-api" Oct 30 13:45:01.985786 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:01.985669 4931 cpu_manager.go:414] "containerMap: removing container" podUID="b1e86eea-4824-4c50-8ad3-2a15c3365c3d" containerName="main" Oct 30 13:45:01.985786 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:01.985674 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1e86eea-4824-4c50-8ad3-2a15c3365c3d" containerName="main" Oct 30 13:45:01.985786 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:01.985683 4931 cpu_manager.go:414] "containerMap: removing container" podUID="3e549a03-6e0a-4068-81f1-8747bc5570a0" containerName="discovery" Oct 30 13:45:01.985786 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:01.985688 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e549a03-6e0a-4068-81f1-8747bc5570a0" containerName="discovery" Oct 30 13:45:01.985786 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:01.985694 4931 cpu_manager.go:414] "containerMap: removing container" podUID="902d392f-4a4d-4ba1-8a6e-7b53c03b6691" containerName="revoke-keys" Oct 30 13:45:01.985786 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:01.985699 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="902d392f-4a4d-4ba1-8a6e-7b53c03b6691" containerName="revoke-keys" Oct 30 13:45:01.985786 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:01.985753 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a504237-e5a3-491f-8f84-0419cf502f9a" containerName="main" Oct 30 13:45:01.985786 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:01.985763 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="996e2b88-ae9e-4e95-8a65-4180de0588b0" containerName="maas-api" Oct 30 13:45:01.985786 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:01.985769 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1e86eea-4824-4c50-8ad3-2a15c3365c3d" containerName="main" Oct 30 13:45:01.985786 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:01.985775 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e549a03-6e0a-4068-81f1-8747bc5570a0" containerName="discovery" Oct 30 13:45:01.985786 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:01.985781 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="902d392f-4a4d-4ba1-8a6e-7b53c03b6691" containerName="revoke-keys" Oct 30 13:45:01.985786 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:01.985789 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3d6937c-4924-4bb7-9449-6c4df4306904" containerName="payload-processing" Oct 30 13:45:01.989120 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:01.989102 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2tbzn" Oct 30 13:45:02.018499 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:02.018468 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:45:02.018701 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:02.018506 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:45:02.018773 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:02.018538 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:45:02.018773 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:02.018566 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:45:02.020894 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:02.020874 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2tbzn"] Oct 30 13:45:02.147462 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:02.147425 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f364dcb-cdc5-4f66-b0a7-8c72033404c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-2tbzn\" (UID: \"5f364dcb-cdc5-4f66-b0a7-8c72033404c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2tbzn" Oct 30 13:45:02.147659 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:02.147545 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pthxv\" (UniqueName: \"kubernetes.io/projected/5f364dcb-cdc5-4f66-b0a7-8c72033404c7-kube-api-access-pthxv\") pod \"maas-api-revoke-sub-2e1f5b5f-2tbzn\" (UID: \"5f364dcb-cdc5-4f66-b0a7-8c72033404c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2tbzn" Oct 30 13:45:02.147659 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:02.147581 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f364dcb-cdc5-4f66-b0a7-8c72033404c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-2tbzn\" (UID: \"5f364dcb-cdc5-4f66-b0a7-8c72033404c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2tbzn" Oct 30 13:45:02.248944 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:02.248854 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f364dcb-cdc5-4f66-b0a7-8c72033404c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-2tbzn\" (UID: \"5f364dcb-cdc5-4f66-b0a7-8c72033404c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2tbzn" Oct 30 13:45:02.249118 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:02.248999 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pthxv\" (UniqueName: \"kubernetes.io/projected/5f364dcb-cdc5-4f66-b0a7-8c72033404c7-kube-api-access-pthxv\") pod \"maas-api-revoke-sub-2e1f5b5f-2tbzn\" (UID: \"5f364dcb-cdc5-4f66-b0a7-8c72033404c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2tbzn" Oct 30 13:45:02.249118 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:02.249030 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f364dcb-cdc5-4f66-b0a7-8c72033404c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-2tbzn\" (UID: \"5f364dcb-cdc5-4f66-b0a7-8c72033404c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2tbzn" Oct 30 13:45:02.249677 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:02.249654 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f364dcb-cdc5-4f66-b0a7-8c72033404c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-2tbzn\" (UID: \"5f364dcb-cdc5-4f66-b0a7-8c72033404c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2tbzn" Oct 30 13:45:02.251186 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:02.251168 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f364dcb-cdc5-4f66-b0a7-8c72033404c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-2tbzn\" (UID: \"5f364dcb-cdc5-4f66-b0a7-8c72033404c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2tbzn" Oct 30 13:45:02.275321 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:02.275293 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pthxv\" (UniqueName: \"kubernetes.io/projected/5f364dcb-cdc5-4f66-b0a7-8c72033404c7-kube-api-access-pthxv\") pod \"maas-api-revoke-sub-2e1f5b5f-2tbzn\" (UID: \"5f364dcb-cdc5-4f66-b0a7-8c72033404c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2tbzn" Oct 30 13:45:02.302464 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:02.302432 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2tbzn" Oct 30 13:45:02.604184 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:02.604154 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2tbzn"] Oct 30 13:45:02.611057 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:45:02.611030 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f364dcb_cdc5_4f66_b0a7_8c72033404c7.slice/crio-39fbd26bd3c8ab3ee41bbcecdbca247c5e052eda6fe6d04b248265d6e4a2d3b6 WatchSource:0}: Error finding container 39fbd26bd3c8ab3ee41bbcecdbca247c5e052eda6fe6d04b248265d6e4a2d3b6: Status 404 returned error can't find the container with id 39fbd26bd3c8ab3ee41bbcecdbca247c5e052eda6fe6d04b248265d6e4a2d3b6 Oct 30 13:45:02.721081 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:02.721050 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/2.log" Oct 30 13:45:02.725852 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:02.725829 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/2.log" Oct 30 13:45:02.841553 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:02.841516 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2tbzn" event={"ID":"5f364dcb-cdc5-4f66-b0a7-8c72033404c7","Type":"ContainerStarted","Data":"39fbd26bd3c8ab3ee41bbcecdbca247c5e052eda6fe6d04b248265d6e4a2d3b6"} Oct 30 13:45:03.849024 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:03.848925 4931 generic.go:356] "Generic (PLEG): container finished" podID="5f364dcb-cdc5-4f66-b0a7-8c72033404c7" containerID="79af1d3339dbf5d5ccdc61cbea53a8a53168e82dd339d7a1bfc783945296b1ed" exitCode=0 Oct 30 13:45:03.849024 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:03.849003 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2tbzn" event={"ID":"5f364dcb-cdc5-4f66-b0a7-8c72033404c7","Type":"ContainerDied","Data":"79af1d3339dbf5d5ccdc61cbea53a8a53168e82dd339d7a1bfc783945296b1ed"} Oct 30 13:45:03.919004 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:03.918962 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl"] Oct 30 13:45:03.922550 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:03.922531 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" Oct 30 13:45:03.955092 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:03.955039 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl"] Oct 30 13:45:04.066341 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.066302 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2n4c\" (UniqueName: \"kubernetes.io/projected/faa03672-ba31-4767-9617-b6e0f5809a1d-kube-api-access-p2n4c\") pod \"maas-default-gateway-openshift-default-58c6585447-m4kcl\" (UID: \"faa03672-ba31-4767-9617-b6e0f5809a1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" Oct 30 13:45:04.066341 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.066349 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa03672-ba31-4767-9617-b6e0f5809a1d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-m4kcl\" (UID: \"faa03672-ba31-4767-9617-b6e0f5809a1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" Oct 30 13:45:04.066563 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.066440 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa03672-ba31-4767-9617-b6e0f5809a1d-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-m4kcl\" (UID: \"faa03672-ba31-4767-9617-b6e0f5809a1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" Oct 30 13:45:04.066563 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.066467 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa03672-ba31-4767-9617-b6e0f5809a1d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-m4kcl\" (UID: \"faa03672-ba31-4767-9617-b6e0f5809a1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" Oct 30 13:45:04.066563 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.066494 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa03672-ba31-4767-9617-b6e0f5809a1d-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-m4kcl\" (UID: \"faa03672-ba31-4767-9617-b6e0f5809a1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" Oct 30 13:45:04.066563 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.066528 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa03672-ba31-4767-9617-b6e0f5809a1d-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-m4kcl\" (UID: \"faa03672-ba31-4767-9617-b6e0f5809a1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" Oct 30 13:45:04.066732 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.066651 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa03672-ba31-4767-9617-b6e0f5809a1d-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-m4kcl\" (UID: \"faa03672-ba31-4767-9617-b6e0f5809a1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" Oct 30 13:45:04.066732 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.066671 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa03672-ba31-4767-9617-b6e0f5809a1d-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-m4kcl\" (UID: \"faa03672-ba31-4767-9617-b6e0f5809a1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" Oct 30 13:45:04.066732 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.066711 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa03672-ba31-4767-9617-b6e0f5809a1d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-m4kcl\" (UID: \"faa03672-ba31-4767-9617-b6e0f5809a1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" Oct 30 13:45:04.167586 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.167505 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa03672-ba31-4767-9617-b6e0f5809a1d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-m4kcl\" (UID: \"faa03672-ba31-4767-9617-b6e0f5809a1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" Oct 30 13:45:04.167586 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.167547 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa03672-ba31-4767-9617-b6e0f5809a1d-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-m4kcl\" (UID: \"faa03672-ba31-4767-9617-b6e0f5809a1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" Oct 30 13:45:04.167586 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.167573 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa03672-ba31-4767-9617-b6e0f5809a1d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-m4kcl\" (UID: \"faa03672-ba31-4767-9617-b6e0f5809a1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" Oct 30 13:45:04.167897 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.167599 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa03672-ba31-4767-9617-b6e0f5809a1d-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-m4kcl\" (UID: \"faa03672-ba31-4767-9617-b6e0f5809a1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" Oct 30 13:45:04.167897 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.167617 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa03672-ba31-4767-9617-b6e0f5809a1d-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-m4kcl\" (UID: \"faa03672-ba31-4767-9617-b6e0f5809a1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" Oct 30 13:45:04.167897 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.167707 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa03672-ba31-4767-9617-b6e0f5809a1d-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-m4kcl\" (UID: \"faa03672-ba31-4767-9617-b6e0f5809a1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" Oct 30 13:45:04.167897 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.167733 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa03672-ba31-4767-9617-b6e0f5809a1d-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-m4kcl\" (UID: \"faa03672-ba31-4767-9617-b6e0f5809a1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" Oct 30 13:45:04.167897 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.167788 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa03672-ba31-4767-9617-b6e0f5809a1d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-m4kcl\" (UID: \"faa03672-ba31-4767-9617-b6e0f5809a1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" Oct 30 13:45:04.167897 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.167833 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2n4c\" (UniqueName: \"kubernetes.io/projected/faa03672-ba31-4767-9617-b6e0f5809a1d-kube-api-access-p2n4c\") pod \"maas-default-gateway-openshift-default-58c6585447-m4kcl\" (UID: \"faa03672-ba31-4767-9617-b6e0f5809a1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" Oct 30 13:45:04.168244 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.168048 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa03672-ba31-4767-9617-b6e0f5809a1d-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-m4kcl\" (UID: \"faa03672-ba31-4767-9617-b6e0f5809a1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" Oct 30 13:45:04.168244 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.168117 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa03672-ba31-4767-9617-b6e0f5809a1d-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-m4kcl\" (UID: \"faa03672-ba31-4767-9617-b6e0f5809a1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" Oct 30 13:45:04.168244 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.168233 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa03672-ba31-4767-9617-b6e0f5809a1d-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-m4kcl\" (UID: \"faa03672-ba31-4767-9617-b6e0f5809a1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" Oct 30 13:45:04.168479 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.168347 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa03672-ba31-4767-9617-b6e0f5809a1d-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-m4kcl\" (UID: \"faa03672-ba31-4767-9617-b6e0f5809a1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" Oct 30 13:45:04.168559 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.168535 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa03672-ba31-4767-9617-b6e0f5809a1d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-m4kcl\" (UID: \"faa03672-ba31-4767-9617-b6e0f5809a1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" Oct 30 13:45:04.169970 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.169950 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa03672-ba31-4767-9617-b6e0f5809a1d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-m4kcl\" (UID: \"faa03672-ba31-4767-9617-b6e0f5809a1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" Oct 30 13:45:04.170486 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.170465 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa03672-ba31-4767-9617-b6e0f5809a1d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-m4kcl\" (UID: \"faa03672-ba31-4767-9617-b6e0f5809a1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" Oct 30 13:45:04.193476 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.193450 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa03672-ba31-4767-9617-b6e0f5809a1d-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-m4kcl\" (UID: \"faa03672-ba31-4767-9617-b6e0f5809a1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" Oct 30 13:45:04.197056 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.197033 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2n4c\" (UniqueName: \"kubernetes.io/projected/faa03672-ba31-4767-9617-b6e0f5809a1d-kube-api-access-p2n4c\") pod \"maas-default-gateway-openshift-default-58c6585447-m4kcl\" (UID: \"faa03672-ba31-4767-9617-b6e0f5809a1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" Oct 30 13:45:04.235934 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.235912 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" Oct 30 13:45:04.554945 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.554916 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl"] Oct 30 13:45:04.556953 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:45:04.556907 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaa03672_ba31_4767_9617_b6e0f5809a1d.slice/crio-e99a244e6d9fa4e3253c201832f3c1e3ef4e5f791c73ee2559765907eb5ae312 WatchSource:0}: Error finding container e99a244e6d9fa4e3253c201832f3c1e3ef4e5f791c73ee2559765907eb5ae312: Status 404 returned error can't find the container with id e99a244e6d9fa4e3253c201832f3c1e3ef4e5f791c73ee2559765907eb5ae312 Oct 30 13:45:04.559477 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.559438 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:45:04.559551 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.559529 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:45:04.559590 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.559576 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:45:04.754698 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.754658 4931 scope.go:122] "RemoveContainer" containerID="90d1076be300e897f627dd862b517a2adf98fbde22aa9ac8ac68696da7ee2f02" Oct 30 13:45:04.766040 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.766015 4931 scope.go:122] "RemoveContainer" containerID="cb7e2ca00c4b75e7bcdffdda4369223cb64ec81bc8a4200dafc8eca887e54902" Oct 30 13:45:04.775749 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.775729 4931 scope.go:122] "RemoveContainer" containerID="dffb818ff7317219a765512d4566285c60c364c9062fca336f3491bb2a458e3d" Oct 30 13:45:04.785868 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.785850 4931 scope.go:122] "RemoveContainer" containerID="e0509e26a8742eef5725f4c7f70b0f9ee0943003fdb2ef25d15209ff4675db67" Oct 30 13:45:04.798312 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.795172 4931 scope.go:122] "RemoveContainer" containerID="31517084b34d3d04199f74fcd5550344b8be3bb3b1810c632f553503c31ebc4d" Oct 30 13:45:04.806773 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.806753 4931 scope.go:122] "RemoveContainer" containerID="f2ad03511ca29a990448840f20a566bea86d7089846c4e48d1508719652fc588" Oct 30 13:45:04.817925 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.817904 4931 scope.go:122] "RemoveContainer" containerID="bf68d3e91f8b6f6e4659290ac9c5be396b2e754ee2a7f410ce091fc5ce952989" Oct 30 13:45:04.828316 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.828294 4931 scope.go:122] "RemoveContainer" containerID="e187ce1424ddb8b4ff7f85f99dac8f235930df4945d56e3e888234425bec65d7" Oct 30 13:45:04.838516 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.838489 4931 scope.go:122] "RemoveContainer" containerID="5811ca0069e1875201718c5e7485e122c0c641f315c11560311efbf54507453f" Oct 30 13:45:04.864688 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.864654 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" event={"ID":"faa03672-ba31-4767-9617-b6e0f5809a1d","Type":"ContainerStarted","Data":"ea4beea8e30f03df124c51210b14430a53846201423fd289907203bdae729e3a"} Oct 30 13:45:04.865089 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.864715 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" event={"ID":"faa03672-ba31-4767-9617-b6e0f5809a1d","Type":"ContainerStarted","Data":"e99a244e6d9fa4e3253c201832f3c1e3ef4e5f791c73ee2559765907eb5ae312"} Oct 30 13:45:04.937351 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:04.936281 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" podStartSLOduration=1.936242196 podStartE2EDuration="1.936242196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:04.933972089 +0000 UTC m=+3603.537470536" watchObservedRunningTime="2026-10-30 13:45:04.936242196 +0000 UTC m=+3603.539740643" Oct 30 13:45:05.027086 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:05.027056 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2tbzn" Oct 30 13:45:05.176767 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:05.176663 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f364dcb-cdc5-4f66-b0a7-8c72033404c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f364dcb-cdc5-4f66-b0a7-8c72033404c7-maas-api-openshift-ca\") pod \"5f364dcb-cdc5-4f66-b0a7-8c72033404c7\" (UID: \"5f364dcb-cdc5-4f66-b0a7-8c72033404c7\") " Oct 30 13:45:05.176767 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:05.176741 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f364dcb-cdc5-4f66-b0a7-8c72033404c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f364dcb-cdc5-4f66-b0a7-8c72033404c7-maas-api-xks-ca\") pod \"5f364dcb-cdc5-4f66-b0a7-8c72033404c7\" (UID: \"5f364dcb-cdc5-4f66-b0a7-8c72033404c7\") " Oct 30 13:45:05.177094 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:05.176846 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f364dcb-cdc5-4f66-b0a7-8c72033404c7-kube-api-access-pthxv\" (UniqueName: \"kubernetes.io/projected/5f364dcb-cdc5-4f66-b0a7-8c72033404c7-kube-api-access-pthxv\") pod \"5f364dcb-cdc5-4f66-b0a7-8c72033404c7\" (UID: \"5f364dcb-cdc5-4f66-b0a7-8c72033404c7\") " Oct 30 13:45:05.177094 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:05.177012 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f364dcb-cdc5-4f66-b0a7-8c72033404c7-maas-api-openshift-ca" pod "5f364dcb-cdc5-4f66-b0a7-8c72033404c7" (UID: "5f364dcb-cdc5-4f66-b0a7-8c72033404c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:05.177213 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:05.177194 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f364dcb-cdc5-4f66-b0a7-8c72033404c7-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:45:05.178886 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:05.178856 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f364dcb-cdc5-4f66-b0a7-8c72033404c7-kube-api-access-pthxv" pod "5f364dcb-cdc5-4f66-b0a7-8c72033404c7" (UID: "5f364dcb-cdc5-4f66-b0a7-8c72033404c7"). InnerVolumeSpecName "kube-api-access-pthxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:05.179008 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:05.178896 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f364dcb-cdc5-4f66-b0a7-8c72033404c7-maas-api-xks-ca" pod "5f364dcb-cdc5-4f66-b0a7-8c72033404c7" (UID: "5f364dcb-cdc5-4f66-b0a7-8c72033404c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:05.253288 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:05.249669 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" Oct 30 13:45:05.257446 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:05.257401 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-m4kcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 30 13:45:05.257628 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:05.257479 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" podUID="faa03672-ba31-4767-9617-b6e0f5809a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 30 13:45:05.278635 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:05.278605 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f364dcb-cdc5-4f66-b0a7-8c72033404c7-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:45:05.278635 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:05.278631 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pthxv\" (UniqueName: \"kubernetes.io/projected/5f364dcb-cdc5-4f66-b0a7-8c72033404c7-kube-api-access-pthxv\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:45:05.872812 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:05.872722 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2tbzn" Oct 30 13:45:05.872812 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:05.872757 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2tbzn" event={"ID":"5f364dcb-cdc5-4f66-b0a7-8c72033404c7","Type":"ContainerDied","Data":"39fbd26bd3c8ab3ee41bbcecdbca247c5e052eda6fe6d04b248265d6e4a2d3b6"} Oct 30 13:45:05.873303 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:05.872814 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39fbd26bd3c8ab3ee41bbcecdbca247c5e052eda6fe6d04b248265d6e4a2d3b6" Oct 30 13:45:06.250297 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:06.237716 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-m4kcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 30 13:45:06.250297 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:06.237799 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" podUID="faa03672-ba31-4767-9617-b6e0f5809a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 30 13:45:07.250291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:07.241682 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-m4kcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 30 13:45:07.250291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:07.241752 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" podUID="faa03672-ba31-4767-9617-b6e0f5809a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 30 13:45:07.380746 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:45:07.380709 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:45:08.238727 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:08.238685 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-m4kcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 30 13:45:08.239050 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:08.239018 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" podUID="faa03672-ba31-4767-9617-b6e0f5809a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 30 13:45:09.236713 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:09.236667 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-m4kcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 30 13:45:09.237370 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:09.237331 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" podUID="faa03672-ba31-4767-9617-b6e0f5809a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 30 13:45:10.257491 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:10.257445 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-m4kcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 30 13:45:10.257916 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:10.257520 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" podUID="faa03672-ba31-4767-9617-b6e0f5809a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 30 13:45:11.236292 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:11.236236 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-m4kcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 30 13:45:11.236550 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:11.236322 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" podUID="faa03672-ba31-4767-9617-b6e0f5809a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 30 13:45:12.737649 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:12.737611 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" Oct 30 13:45:12.738135 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:12.738070 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" Oct 30 13:45:12.739011 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:12.738990 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl" Oct 30 13:45:13.257549 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:13.257506 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fwl5"] Oct 30 13:45:13.257928 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:13.257913 4931 cpu_manager.go:414] "containerMap: removing container" podUID="5f364dcb-cdc5-4f66-b0a7-8c72033404c7" containerName="revoke-keys" Oct 30 13:45:13.258019 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:13.257930 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f364dcb-cdc5-4f66-b0a7-8c72033404c7" containerName="revoke-keys" Oct 30 13:45:13.258083 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:13.258049 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f364dcb-cdc5-4f66-b0a7-8c72033404c7" containerName="revoke-keys" Oct 30 13:45:13.261400 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:13.261376 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fwl5" Oct 30 13:45:13.282816 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:13.282783 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:45:13.282944 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:13.282853 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:45:13.282944 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:13.282872 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:45:13.283065 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:13.282958 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:45:13.285845 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:13.285822 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fwl5"] Oct 30 13:45:13.299903 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:13.299872 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:45:13.355892 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:13.355856 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/558d636e-5349-4555-8a27-c6cd7a81021b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-8fwl5\" (UID: \"558d636e-5349-4555-8a27-c6cd7a81021b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fwl5" Oct 30 13:45:13.356057 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:13.355911 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/558d636e-5349-4555-8a27-c6cd7a81021b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-8fwl5\" (UID: \"558d636e-5349-4555-8a27-c6cd7a81021b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fwl5" Oct 30 13:45:13.356057 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:13.356040 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb55l\" (UniqueName: \"kubernetes.io/projected/558d636e-5349-4555-8a27-c6cd7a81021b-kube-api-access-kb55l\") pod \"maas-api-revoke-sub-ec843018-8fwl5\" (UID: \"558d636e-5349-4555-8a27-c6cd7a81021b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fwl5" Oct 30 13:45:13.456717 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:13.456683 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/558d636e-5349-4555-8a27-c6cd7a81021b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-8fwl5\" (UID: \"558d636e-5349-4555-8a27-c6cd7a81021b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fwl5" Oct 30 13:45:13.456885 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:13.456735 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/558d636e-5349-4555-8a27-c6cd7a81021b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-8fwl5\" (UID: \"558d636e-5349-4555-8a27-c6cd7a81021b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fwl5" Oct 30 13:45:13.456885 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:13.456767 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb55l\" (UniqueName: \"kubernetes.io/projected/558d636e-5349-4555-8a27-c6cd7a81021b-kube-api-access-kb55l\") pod \"maas-api-revoke-sub-ec843018-8fwl5\" (UID: \"558d636e-5349-4555-8a27-c6cd7a81021b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fwl5" Oct 30 13:45:13.457639 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:13.457607 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/558d636e-5349-4555-8a27-c6cd7a81021b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-8fwl5\" (UID: \"558d636e-5349-4555-8a27-c6cd7a81021b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fwl5" Oct 30 13:45:13.459496 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:13.459475 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/558d636e-5349-4555-8a27-c6cd7a81021b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-8fwl5\" (UID: \"558d636e-5349-4555-8a27-c6cd7a81021b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fwl5" Oct 30 13:45:13.482549 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:13.482527 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb55l\" (UniqueName: \"kubernetes.io/projected/558d636e-5349-4555-8a27-c6cd7a81021b-kube-api-access-kb55l\") pod \"maas-api-revoke-sub-ec843018-8fwl5\" (UID: \"558d636e-5349-4555-8a27-c6cd7a81021b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fwl5" Oct 30 13:45:13.575429 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:13.575340 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fwl5" Oct 30 13:45:13.903430 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:13.903392 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fwl5"] Oct 30 13:45:13.904717 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:45:13.904687 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod558d636e_5349_4555_8a27_c6cd7a81021b.slice/crio-6683016ae81aa0733688bcf1a7bf9dcbcce2b5230d8e9cbdebe4e3eebb6a7568 WatchSource:0}: Error finding container 6683016ae81aa0733688bcf1a7bf9dcbcce2b5230d8e9cbdebe4e3eebb6a7568: Status 404 returned error can't find the container with id 6683016ae81aa0733688bcf1a7bf9dcbcce2b5230d8e9cbdebe4e3eebb6a7568 Oct 30 13:45:13.921990 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:13.921950 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fwl5" event={"ID":"558d636e-5349-4555-8a27-c6cd7a81021b","Type":"ContainerStarted","Data":"6683016ae81aa0733688bcf1a7bf9dcbcce2b5230d8e9cbdebe4e3eebb6a7568"} Oct 30 13:45:14.930213 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:14.930173 4931 generic.go:356] "Generic (PLEG): container finished" podID="558d636e-5349-4555-8a27-c6cd7a81021b" containerID="c933e54f5a12d672c0fa19b465d2b5a414f70e2984ec1e43cd72130290269673" exitCode=0 Oct 30 13:45:14.930642 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:14.930209 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fwl5" event={"ID":"558d636e-5349-4555-8a27-c6cd7a81021b","Type":"ContainerDied","Data":"c933e54f5a12d672c0fa19b465d2b5a414f70e2984ec1e43cd72130290269673"} Oct 30 13:45:16.078956 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:16.078929 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fwl5" Oct 30 13:45:16.184687 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:16.184654 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/558d636e-5349-4555-8a27-c6cd7a81021b-kube-api-access-kb55l\" (UniqueName: \"kubernetes.io/projected/558d636e-5349-4555-8a27-c6cd7a81021b-kube-api-access-kb55l\") pod \"558d636e-5349-4555-8a27-c6cd7a81021b\" (UID: \"558d636e-5349-4555-8a27-c6cd7a81021b\") " Oct 30 13:45:16.184910 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:16.184812 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/558d636e-5349-4555-8a27-c6cd7a81021b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/558d636e-5349-4555-8a27-c6cd7a81021b-maas-api-xks-ca\") pod \"558d636e-5349-4555-8a27-c6cd7a81021b\" (UID: \"558d636e-5349-4555-8a27-c6cd7a81021b\") " Oct 30 13:45:16.184910 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:16.184849 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/558d636e-5349-4555-8a27-c6cd7a81021b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/558d636e-5349-4555-8a27-c6cd7a81021b-maas-api-openshift-ca\") pod \"558d636e-5349-4555-8a27-c6cd7a81021b\" (UID: \"558d636e-5349-4555-8a27-c6cd7a81021b\") " Oct 30 13:45:16.185364 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:16.185331 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/558d636e-5349-4555-8a27-c6cd7a81021b-maas-api-openshift-ca" pod "558d636e-5349-4555-8a27-c6cd7a81021b" (UID: "558d636e-5349-4555-8a27-c6cd7a81021b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:16.187008 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:16.186953 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/558d636e-5349-4555-8a27-c6cd7a81021b-maas-api-xks-ca" pod "558d636e-5349-4555-8a27-c6cd7a81021b" (UID: "558d636e-5349-4555-8a27-c6cd7a81021b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:16.187097 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:16.187050 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/558d636e-5349-4555-8a27-c6cd7a81021b-kube-api-access-kb55l" pod "558d636e-5349-4555-8a27-c6cd7a81021b" (UID: "558d636e-5349-4555-8a27-c6cd7a81021b"). InnerVolumeSpecName "kube-api-access-kb55l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:16.285804 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:16.285760 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/558d636e-5349-4555-8a27-c6cd7a81021b-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:45:16.285804 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:16.285800 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb55l\" (UniqueName: \"kubernetes.io/projected/558d636e-5349-4555-8a27-c6cd7a81021b-kube-api-access-kb55l\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:45:16.285804 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:16.285814 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/558d636e-5349-4555-8a27-c6cd7a81021b-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:45:16.944482 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:16.944442 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fwl5" event={"ID":"558d636e-5349-4555-8a27-c6cd7a81021b","Type":"ContainerDied","Data":"6683016ae81aa0733688bcf1a7bf9dcbcce2b5230d8e9cbdebe4e3eebb6a7568"} Oct 30 13:45:16.944696 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:16.944509 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6683016ae81aa0733688bcf1a7bf9dcbcce2b5230d8e9cbdebe4e3eebb6a7568" Oct 30 13:45:16.944696 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:16.944455 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fwl5" Oct 30 13:45:18.983527 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:18.983493 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9"] Oct 30 13:45:18.983975 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:18.983884 4931 cpu_manager.go:414] "containerMap: removing container" podUID="558d636e-5349-4555-8a27-c6cd7a81021b" containerName="revoke-keys" Oct 30 13:45:18.983975 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:18.983895 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="558d636e-5349-4555-8a27-c6cd7a81021b" containerName="revoke-keys" Oct 30 13:45:18.983975 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:18.983967 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="558d636e-5349-4555-8a27-c6cd7a81021b" containerName="revoke-keys" Oct 30 13:45:18.987275 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:18.987247 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" Oct 30 13:45:19.010286 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:19.010245 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27bc72a1-afb4-4124-a6d6-95d6c1f0eb48-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-4vfv9\" (UID: \"27bc72a1-afb4-4124-a6d6-95d6c1f0eb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" Oct 30 13:45:19.010427 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:19.010366 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27bc72a1-afb4-4124-a6d6-95d6c1f0eb48-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-4vfv9\" (UID: \"27bc72a1-afb4-4124-a6d6-95d6c1f0eb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" Oct 30 13:45:19.010480 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:19.010464 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27bc72a1-afb4-4124-a6d6-95d6c1f0eb48-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-4vfv9\" (UID: \"27bc72a1-afb4-4124-a6d6-95d6c1f0eb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" Oct 30 13:45:19.010539 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:19.010495 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27bc72a1-afb4-4124-a6d6-95d6c1f0eb48-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-4vfv9\" (UID: \"27bc72a1-afb4-4124-a6d6-95d6c1f0eb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" Oct 30 13:45:19.010539 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:19.010522 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27bc72a1-afb4-4124-a6d6-95d6c1f0eb48-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-4vfv9\" (UID: \"27bc72a1-afb4-4124-a6d6-95d6c1f0eb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" Oct 30 13:45:19.010670 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:19.010612 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27bc72a1-afb4-4124-a6d6-95d6c1f0eb48-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-4vfv9\" (UID: \"27bc72a1-afb4-4124-a6d6-95d6c1f0eb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" Oct 30 13:45:19.010670 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:19.010660 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27bc72a1-afb4-4124-a6d6-95d6c1f0eb48-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-4vfv9\" (UID: \"27bc72a1-afb4-4124-a6d6-95d6c1f0eb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" Oct 30 13:45:19.010824 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:19.010710 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtbxq\" (UniqueName: \"kubernetes.io/projected/27bc72a1-afb4-4124-a6d6-95d6c1f0eb48-kube-api-access-dtbxq\") pod \"maas-default-gateway-openshift-default-58c6585447-4vfv9\" (UID: \"27bc72a1-afb4-4124-a6d6-95d6c1f0eb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" Oct 30 13:45:19.010824 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:19.010758 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27bc72a1-afb4-4124-a6d6-95d6c1f0eb48-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-4vfv9\" (UID: \"27bc72a1-afb4-4124-a6d6-95d6c1f0eb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" Oct 30 13:45:19.027861 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:19.027822 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9"] Oct 30 13:45:19.111910 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:19.111879 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27bc72a1-afb4-4124-a6d6-95d6c1f0eb48-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-4vfv9\" (UID: \"27bc72a1-afb4-4124-a6d6-95d6c1f0eb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" Oct 30 13:45:19.112092 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:19.111919 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27bc72a1-afb4-4124-a6d6-95d6c1f0eb48-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-4vfv9\" (UID: \"27bc72a1-afb4-4124-a6d6-95d6c1f0eb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" Oct 30 13:45:19.112092 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:19.111939 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtbxq\" (UniqueName: \"kubernetes.io/projected/27bc72a1-afb4-4124-a6d6-95d6c1f0eb48-kube-api-access-dtbxq\") pod \"maas-default-gateway-openshift-default-58c6585447-4vfv9\" (UID: \"27bc72a1-afb4-4124-a6d6-95d6c1f0eb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" Oct 30 13:45:19.112092 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:19.111992 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27bc72a1-afb4-4124-a6d6-95d6c1f0eb48-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-4vfv9\" (UID: \"27bc72a1-afb4-4124-a6d6-95d6c1f0eb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" Oct 30 13:45:19.112092 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:19.112021 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27bc72a1-afb4-4124-a6d6-95d6c1f0eb48-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-4vfv9\" (UID: \"27bc72a1-afb4-4124-a6d6-95d6c1f0eb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" Oct 30 13:45:19.112092 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:19.112052 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27bc72a1-afb4-4124-a6d6-95d6c1f0eb48-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-4vfv9\" (UID: \"27bc72a1-afb4-4124-a6d6-95d6c1f0eb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" Oct 30 13:45:19.112092 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:19.112094 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27bc72a1-afb4-4124-a6d6-95d6c1f0eb48-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-4vfv9\" (UID: \"27bc72a1-afb4-4124-a6d6-95d6c1f0eb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" Oct 30 13:45:19.112500 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:19.112120 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27bc72a1-afb4-4124-a6d6-95d6c1f0eb48-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-4vfv9\" (UID: \"27bc72a1-afb4-4124-a6d6-95d6c1f0eb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" Oct 30 13:45:19.112500 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:19.112157 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27bc72a1-afb4-4124-a6d6-95d6c1f0eb48-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-4vfv9\" (UID: \"27bc72a1-afb4-4124-a6d6-95d6c1f0eb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" Oct 30 13:45:19.112500 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:19.112357 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27bc72a1-afb4-4124-a6d6-95d6c1f0eb48-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-4vfv9\" (UID: \"27bc72a1-afb4-4124-a6d6-95d6c1f0eb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" Oct 30 13:45:19.112709 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:19.112688 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27bc72a1-afb4-4124-a6d6-95d6c1f0eb48-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-4vfv9\" (UID: \"27bc72a1-afb4-4124-a6d6-95d6c1f0eb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" Oct 30 13:45:19.112898 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:19.112870 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27bc72a1-afb4-4124-a6d6-95d6c1f0eb48-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-4vfv9\" (UID: \"27bc72a1-afb4-4124-a6d6-95d6c1f0eb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" Oct 30 13:45:19.112969 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:19.112933 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27bc72a1-afb4-4124-a6d6-95d6c1f0eb48-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-4vfv9\" (UID: \"27bc72a1-afb4-4124-a6d6-95d6c1f0eb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" Oct 30 13:45:19.113131 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:19.113110 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27bc72a1-afb4-4124-a6d6-95d6c1f0eb48-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-4vfv9\" (UID: \"27bc72a1-afb4-4124-a6d6-95d6c1f0eb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" Oct 30 13:45:19.114279 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:19.114251 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27bc72a1-afb4-4124-a6d6-95d6c1f0eb48-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-4vfv9\" (UID: \"27bc72a1-afb4-4124-a6d6-95d6c1f0eb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" Oct 30 13:45:19.114900 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:19.114879 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27bc72a1-afb4-4124-a6d6-95d6c1f0eb48-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-4vfv9\" (UID: \"27bc72a1-afb4-4124-a6d6-95d6c1f0eb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" Oct 30 13:45:19.138393 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:19.138366 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27bc72a1-afb4-4124-a6d6-95d6c1f0eb48-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-4vfv9\" (UID: \"27bc72a1-afb4-4124-a6d6-95d6c1f0eb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" Oct 30 13:45:19.138603 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:19.138585 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtbxq\" (UniqueName: \"kubernetes.io/projected/27bc72a1-afb4-4124-a6d6-95d6c1f0eb48-kube-api-access-dtbxq\") pod \"maas-default-gateway-openshift-default-58c6585447-4vfv9\" (UID: \"27bc72a1-afb4-4124-a6d6-95d6c1f0eb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" Oct 30 13:45:19.301797 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:19.301705 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" Oct 30 13:45:19.638750 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:19.638721 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9"] Oct 30 13:45:19.641042 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:45:19.641016 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27bc72a1_afb4_4124_a6d6_95d6c1f0eb48.slice/crio-9172da085879324a957fe0f2ca2e3ad2dad55901a6263f473e7733a94a2e17f7 WatchSource:0}: Error finding container 9172da085879324a957fe0f2ca2e3ad2dad55901a6263f473e7733a94a2e17f7: Status 404 returned error can't find the container with id 9172da085879324a957fe0f2ca2e3ad2dad55901a6263f473e7733a94a2e17f7 Oct 30 13:45:19.643917 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:19.643886 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:45:19.644000 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:19.643948 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:45:19.644000 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:19.643976 4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:45:19.971364 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:19.971312 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" event={"ID":"27bc72a1-afb4-4124-a6d6-95d6c1f0eb48","Type":"ContainerStarted","Data":"9ef2c4c39c3a5b6d0fea135aa9f55ef15f6c8b3d6f2c9c176d112d2e40f03056"} Oct 30 13:45:19.971581 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:19.971408 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" event={"ID":"27bc72a1-afb4-4124-a6d6-95d6c1f0eb48","Type":"ContainerStarted","Data":"9172da085879324a957fe0f2ca2e3ad2dad55901a6263f473e7733a94a2e17f7"} Oct 30 13:45:20.035021 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:20.034950 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" podStartSLOduration=2.034936189 podStartE2EDuration="2.034936189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:20.034708325 +0000 UTC m=+3618.638206771" watchObservedRunningTime="2026-10-30 13:45:20.034936189 +0000 UTC m=+3618.638434635" Oct 30 13:45:20.302936 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:20.302840 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" Oct 30 13:45:20.304547 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:20.304516 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-4vfv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body= Oct 30 13:45:20.304687 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:20.304568 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" podUID="27bc72a1-afb4-4124-a6d6-95d6c1f0eb48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" Oct 30 13:45:21.302881 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:21.302835 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-4vfv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body= Oct 30 13:45:21.303424 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:21.302904 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" podUID="27bc72a1-afb4-4124-a6d6-95d6c1f0eb48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" Oct 30 13:45:22.302193 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:22.302150 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-4vfv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body= Oct 30 13:45:22.302417 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:22.302212 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" podUID="27bc72a1-afb4-4124-a6d6-95d6c1f0eb48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" Oct 30 13:45:23.054869 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:23.054801 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de97761d-tf5v5"] Oct 30 13:45:23.060286 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:23.060232 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de97761d-tf5v5"] Oct 30 13:45:23.302465 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:23.302431 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-4vfv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body= Oct 30 13:45:23.302656 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:23.302481 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" podUID="27bc72a1-afb4-4124-a6d6-95d6c1f0eb48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" Oct 30 13:45:23.511943 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:23.511907 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="145f3cc9-6c5a-4c88-bc89-da9c956c224d" path="/var/lib/kubelet/pods/145f3cc9-6c5a-4c88-bc89-da9c956c224d/volumes" Oct 30 13:45:24.302377 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:24.302338 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-4vfv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body= Oct 30 13:45:24.302809 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:24.302398 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" podUID="27bc72a1-afb4-4124-a6d6-95d6c1f0eb48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" Oct 30 13:45:24.377499 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:24.377462 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9"] Oct 30 13:45:24.377691 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:24.377668 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9" podUID="d03b10bb-ca2a-4636-ad2b-69972fbb56cf" containerName="discovery" containerID="cri-o://e91da7006bc29f499d0c4d01085170f6ec6be0845540024581d352a36e29ba09" gracePeriod=30 Oct 30 13:45:24.682156 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:24.682126 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9" Oct 30 13:45:24.789109 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:24.789068 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-local-certs\") pod \"d03b10bb-ca2a-4636-ad2b-69972fbb56cf\" (UID: \"d03b10bb-ca2a-4636-ad2b-69972fbb56cf\") " Oct 30 13:45:24.789109 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:24.789109 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-kube-api-access-mnzdj\" (UniqueName: \"kubernetes.io/projected/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-kube-api-access-mnzdj\") pod \"d03b10bb-ca2a-4636-ad2b-69972fbb56cf\" (UID: \"d03b10bb-ca2a-4636-ad2b-69972fbb56cf\") " Oct 30 13:45:24.789436 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:24.789132 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-istio-kubeconfig\") pod \"d03b10bb-ca2a-4636-ad2b-69972fbb56cf\" (UID: \"d03b10bb-ca2a-4636-ad2b-69972fbb56cf\") " Oct 30 13:45:24.789436 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:24.789153 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-istio-csr-dns-cert\") pod \"d03b10bb-ca2a-4636-ad2b-69972fbb56cf\" (UID: \"d03b10bb-ca2a-4636-ad2b-69972fbb56cf\") " Oct 30 13:45:24.789436 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:24.789178 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-cacerts\" (UniqueName: \"kubernetes.io/secret/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-cacerts\") pod \"d03b10bb-ca2a-4636-ad2b-69972fbb56cf\" (UID: \"d03b10bb-ca2a-4636-ad2b-69972fbb56cf\") " Oct 30 13:45:24.789436 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:24.789323 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-istio-token\" (UniqueName: \"kubernetes.io/projected/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-istio-token\") pod \"d03b10bb-ca2a-4636-ad2b-69972fbb56cf\" (UID: \"d03b10bb-ca2a-4636-ad2b-69972fbb56cf\") " Oct 30 13:45:24.789651 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:24.789450 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-istio-csr-ca-configmap\") pod \"d03b10bb-ca2a-4636-ad2b-69972fbb56cf\" (UID: \"d03b10bb-ca2a-4636-ad2b-69972fbb56cf\") " Oct 30 13:45:24.790240 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:24.790209 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-istio-csr-ca-configmap" pod "d03b10bb-ca2a-4636-ad2b-69972fbb56cf" (UID: "d03b10bb-ca2a-4636-ad2b-69972fbb56cf"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:24.791560 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:24.791531 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-local-certs" pod "d03b10bb-ca2a-4636-ad2b-69972fbb56cf" (UID: "d03b10bb-ca2a-4636-ad2b-69972fbb56cf"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:24.791655 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:24.791601 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-cacerts" pod "d03b10bb-ca2a-4636-ad2b-69972fbb56cf" (UID: "d03b10bb-ca2a-4636-ad2b-69972fbb56cf"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:24.791946 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:24.791922 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-istio-kubeconfig" pod "d03b10bb-ca2a-4636-ad2b-69972fbb56cf" (UID: "d03b10bb-ca2a-4636-ad2b-69972fbb56cf"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:24.792061 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:24.792034 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-istio-token" pod "d03b10bb-ca2a-4636-ad2b-69972fbb56cf" (UID: "d03b10bb-ca2a-4636-ad2b-69972fbb56cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:24.792230 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:24.792213 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-istio-csr-dns-cert" pod "d03b10bb-ca2a-4636-ad2b-69972fbb56cf" (UID: "d03b10bb-ca2a-4636-ad2b-69972fbb56cf"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:24.792312 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:24.792261 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-kube-api-access-mnzdj" pod "d03b10bb-ca2a-4636-ad2b-69972fbb56cf" (UID: "d03b10bb-ca2a-4636-ad2b-69972fbb56cf"). InnerVolumeSpecName "kube-api-access-mnzdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:24.891007 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:24.890915 4931 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-local-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:45:24.891007 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:24.890951 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnzdj\" (UniqueName: \"kubernetes.io/projected/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-kube-api-access-mnzdj\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:45:24.891007 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:24.890963 4931 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-istio-kubeconfig\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:45:24.891007 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:24.890973 4931 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-istio-csr-dns-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:45:24.891007 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:24.890982 4931 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-cacerts\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:45:24.891007 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:24.890990 4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:45:24.891007 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:24.890999 4931 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d03b10bb-ca2a-4636-ad2b-69972fbb56cf-istio-csr-ca-configmap\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:45:25.004056 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:25.001978 4931 generic.go:356] "Generic (PLEG): container finished" podID="d03b10bb-ca2a-4636-ad2b-69972fbb56cf" containerID="e91da7006bc29f499d0c4d01085170f6ec6be0845540024581d352a36e29ba09" exitCode=0 Oct 30 13:45:25.004056 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:25.002309 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9" event={"ID":"d03b10bb-ca2a-4636-ad2b-69972fbb56cf","Type":"ContainerDied","Data":"e91da7006bc29f499d0c4d01085170f6ec6be0845540024581d352a36e29ba09"} Oct 30 13:45:25.004056 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:25.002460 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9" event={"ID":"d03b10bb-ca2a-4636-ad2b-69972fbb56cf","Type":"ContainerDied","Data":"3e47d4048573b91110d8c677e24ff8433411add7f460d7ed43b4228dbd7b930d"} Oct 30 13:45:25.004056 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:25.002490 4931 scope.go:122] "RemoveContainer" containerID="e91da7006bc29f499d0c4d01085170f6ec6be0845540024581d352a36e29ba09" Oct 30 13:45:25.004056 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:25.002545 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9" Oct 30 13:45:25.021948 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:25.021784 4931 scope.go:122] "RemoveContainer" containerID="e91da7006bc29f499d0c4d01085170f6ec6be0845540024581d352a36e29ba09" Oct 30 13:45:25.022180 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:45:25.022143 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e91da7006bc29f499d0c4d01085170f6ec6be0845540024581d352a36e29ba09\": container with ID starting with e91da7006bc29f499d0c4d01085170f6ec6be0845540024581d352a36e29ba09 not found: ID does not exist" containerID="e91da7006bc29f499d0c4d01085170f6ec6be0845540024581d352a36e29ba09" Oct 30 13:45:25.022295 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:25.022183 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e91da7006bc29f499d0c4d01085170f6ec6be0845540024581d352a36e29ba09"} err="failed to get container status \"e91da7006bc29f499d0c4d01085170f6ec6be0845540024581d352a36e29ba09\": rpc error: code = NotFound desc = could not find container \"e91da7006bc29f499d0c4d01085170f6ec6be0845540024581d352a36e29ba09\": container with ID starting with e91da7006bc29f499d0c4d01085170f6ec6be0845540024581d352a36e29ba09 not found: ID does not exist" Oct 30 13:45:25.029691 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:25.029660 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42777118-c7bpv"] Oct 30 13:45:25.058563 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:25.058526 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42777118-c7bpv"] Oct 30 13:45:25.080427 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:25.080391 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9"] Oct 30 13:45:25.085213 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:25.085186 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9"] Oct 30 13:45:25.302733 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:25.302688 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-4vfv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body= Oct 30 13:45:25.303149 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:25.302757 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" podUID="27bc72a1-afb4-4124-a6d6-95d6c1f0eb48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" Oct 30 13:45:25.511812 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:25.511776 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b18b275-a794-4416-9bbd-81b153e4e98d" path="/var/lib/kubelet/pods/0b18b275-a794-4416-9bbd-81b153e4e98d/volumes" Oct 30 13:45:25.512229 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:25.512215 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d03b10bb-ca2a-4636-ad2b-69972fbb56cf" path="/var/lib/kubelet/pods/d03b10bb-ca2a-4636-ad2b-69972fbb56cf/volumes" Oct 30 13:45:26.040124 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:26.040086 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5aa264b-ntw2b"] Oct 30 13:45:26.044023 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:26.043989 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5aa264b-ntw2b"] Oct 30 13:45:26.302438 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:26.302343 4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-4vfv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body= Oct 30 13:45:26.302438 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:26.302416 4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" podUID="27bc72a1-afb4-4124-a6d6-95d6c1f0eb48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" Oct 30 13:45:27.512917 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:27.512879 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca29257f-a115-4ff7-b977-9f8c488d6db1" path="/var/lib/kubelet/pods/ca29257f-a115-4ff7-b977-9f8c488d6db1/volumes" Oct 30 13:45:27.986790 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:27.986754 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" Oct 30 13:45:27.987152 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:27.987125 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" Oct 30 13:45:27.987963 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:27.987941 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9" Oct 30 13:45:29.113843 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:29.113811 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:45:37.351644 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:45:37.351598 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:45:47.051814 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:47.051730 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44e71584-xlwx7"] Oct 30 13:45:47.065105 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:47.065069 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44e71584-xlwx7"] Oct 30 13:45:47.511440 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:47.511407 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9e56860-9840-4bd5-bf11-104f28578afe" path="/var/lib/kubelet/pods/f9e56860-9840-4bd5-bf11-104f28578afe/volumes" Oct 30 13:45:52.730330 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:45:52.730284 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:46:03.099360 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:46:03.099323 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:46:04.909170 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:46:04.909138 4931 scope.go:122] "RemoveContainer" containerID="6c8d4ffce367e5f54868df833edbf1159e555ee9cd00cdf6c1fe87da2d29c71f" Oct 30 13:46:04.918366 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:46:04.918337 4931 scope.go:122] "RemoveContainer" containerID="da3fe135651e4337b424c89363e9515865ad08e09a725a678d68080d32228d95" Oct 30 13:46:04.930424 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:46:04.930404 4931 scope.go:122] "RemoveContainer" containerID="92a2ae4dd38ea66b970b8365af6b30d2f2834bf3e25a4d8c57b2e82b44ccd3ce" Oct 30 13:46:04.940541 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:46:04.940513 4931 scope.go:122] "RemoveContainer" containerID="b8cc47f72b5a8fa9e7e8321ab9ea3ff124774cd05722e070037a1f9af57c3585" Oct 30 13:46:06.039077 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:46:06.039045 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e1b470-jqbrm"] Oct 30 13:46:06.051005 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:46:06.050978 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e1b470-jqbrm"] Oct 30 13:46:07.351486 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:46:07.351444 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:46:07.511599 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:46:07.511568 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb8000b8-760a-42bb-9c19-6fdc97354a6a" path="/var/lib/kubelet/pods/fb8000b8-760a-42bb-9c19-6fdc97354a6a/volumes" Oct 30 13:46:07.526724 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:46:07.526696 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:46:24.802921 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:46:24.802872 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:46:29.548305 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:46:29.548245 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:46:37.351555 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:46:37.351521 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:46:46.333385 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:46:46.333345 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:46:51.203746 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:46:51.203705 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:47:04.993500 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:47:04.993469 4931 scope.go:122] "RemoveContainer" containerID="ff45ce4d54d46c427a672b760af586d2561fbb8685325031f2f349c4d79128e2" Oct 30 13:47:05.002443 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:47:05.002422 4931 scope.go:122] "RemoveContainer" containerID="a25b0f2a4a9e72c2e954925993cbedecb3b64b5499a34bb159c50d0325190aa0" Oct 30 13:47:05.010545 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:47:05.010522 4931 scope.go:122] "RemoveContainer" containerID="16c22459193205128ca577a25a1a21b68860060607bf11eb6a12dbebbc463b7a" Oct 30 13:47:05.018422 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:47:05.018402 4931 scope.go:122] "RemoveContainer" containerID="12ba743ef8521cf4c4dc8e898b6c2ef1cc496711a17b8888fbfae65d81e729ef" Oct 30 13:47:05.025713 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:47:05.025696 4931 scope.go:122] "RemoveContainer" containerID="aa188db15d3972e2f64cc5cc579db8291dba17852c16a8dc84637ec4bad65d5f" Oct 30 13:47:05.033512 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:47:05.033494 4931 scope.go:122] "RemoveContainer" containerID="f01a91434e52069c23159dc6c2cda66f5810b0cf54b393ffddc4270e0cc0d0ad" Oct 30 13:47:05.040899 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:47:05.040879 4931 scope.go:122] "RemoveContainer" containerID="3d7b7e97642f0106b1e2529dab5b1bc79466f0ea47b9a068cbf391ac9117dc47" Oct 30 13:47:05.048169 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:47:05.048154 4931 scope.go:122] "RemoveContainer" containerID="6c0ce9d0cfdbf295cb012f5a343eb19f634e1fb710b08a9388906920ca51ed4d" Oct 30 13:47:07.355626 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:47:07.355585 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:47:07.532895 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:47:07.532858 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:47:24.683491 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:47:24.683452 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:47:35.341257 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:47:35.341219 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:47:37.416852 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:47:37.416813 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:47:40.330546 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:47:40.330511 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:48:05.095207 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:05.095172 4931 scope.go:122] "RemoveContainer" containerID="b76c6ec384e3beec392a7e76caaf07513a03c72ca99e2aee847ebd4c4073c137" Oct 30 13:48:05.104066 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:05.104042 4931 scope.go:122] "RemoveContainer" containerID="d978a199f5e9343225a83c1bc228d57230663079f765740623fa79b22f68d92d" Oct 30 13:48:07.350916 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:48:07.350874 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:48:27.910388 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:27.910342 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:48:37.351855 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:48:37.351813 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:48:39.696092 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:39.696052 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt"] Oct 30 13:48:39.696729 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:39.696634 4931 cpu_manager.go:414] "containerMap: removing container" podUID="d03b10bb-ca2a-4636-ad2b-69972fbb56cf" containerName="discovery" Oct 30 13:48:39.696729 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:39.696658 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d03b10bb-ca2a-4636-ad2b-69972fbb56cf" containerName="discovery" Oct 30 13:48:39.696869 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:39.696786 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="d03b10bb-ca2a-4636-ad2b-69972fbb56cf" containerName="discovery" Oct 30 13:48:39.699862 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:39.699839 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt" Oct 30 13:48:39.728966 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:39.728935 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-f53955\"/\"default-dockercfg-pbb7x\"" Oct 30 13:48:39.729144 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:39.729037 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-f53955\"/\"simulator-single-main-f53955-kserve-self-signed-certs\"" Oct 30 13:48:39.730756 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:39.730733 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-f53955\"/\"openshift-service-ca.crt\"" Oct 30 13:48:39.730858 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:39.730779 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-f53955\"/\"kube-root-ca.crt\"" Oct 30 13:48:39.731938 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:39.731919 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt"] Oct 30 13:48:39.803260 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:39.803229 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-tmp-dir\") pod \"simulator-single-main-f53955-kserve-86b865b85c-b99zt\" (UID: \"d4ce772d-7c9b-489b-92e9-6d0999f0c9ad\") " pod="e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt" Oct 30 13:48:39.803444 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:39.803394 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-model-cache\") pod \"simulator-single-main-f53955-kserve-86b865b85c-b99zt\" (UID: \"d4ce772d-7c9b-489b-92e9-6d0999f0c9ad\") " pod="e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt" Oct 30 13:48:39.803444 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:39.803434 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljgqv\" (UniqueName: \"kubernetes.io/projected/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-kube-api-access-ljgqv\") pod \"simulator-single-main-f53955-kserve-86b865b85c-b99zt\" (UID: \"d4ce772d-7c9b-489b-92e9-6d0999f0c9ad\") " pod="e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt" Oct 30 13:48:39.803554 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:39.803522 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-home\") pod \"simulator-single-main-f53955-kserve-86b865b85c-b99zt\" (UID: \"d4ce772d-7c9b-489b-92e9-6d0999f0c9ad\") " pod="e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt" Oct 30 13:48:39.803554 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:39.803546 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-tls-certs\") pod \"simulator-single-main-f53955-kserve-86b865b85c-b99zt\" (UID: \"d4ce772d-7c9b-489b-92e9-6d0999f0c9ad\") " pod="e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt" Oct 30 13:48:39.803641 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:39.803568 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-dshm\") pod \"simulator-single-main-f53955-kserve-86b865b85c-b99zt\" (UID: \"d4ce772d-7c9b-489b-92e9-6d0999f0c9ad\") " pod="e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt" Oct 30 13:48:39.905057 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:39.905024 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-model-cache\") pod \"simulator-single-main-f53955-kserve-86b865b85c-b99zt\" (UID: \"d4ce772d-7c9b-489b-92e9-6d0999f0c9ad\") " pod="e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt" Oct 30 13:48:39.905294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:39.905069 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljgqv\" (UniqueName: \"kubernetes.io/projected/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-kube-api-access-ljgqv\") pod \"simulator-single-main-f53955-kserve-86b865b85c-b99zt\" (UID: \"d4ce772d-7c9b-489b-92e9-6d0999f0c9ad\") " pod="e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt" Oct 30 13:48:39.905294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:39.905105 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-home\") pod \"simulator-single-main-f53955-kserve-86b865b85c-b99zt\" (UID: \"d4ce772d-7c9b-489b-92e9-6d0999f0c9ad\") " pod="e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt" Oct 30 13:48:39.905294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:39.905133 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-tls-certs\") pod \"simulator-single-main-f53955-kserve-86b865b85c-b99zt\" (UID: \"d4ce772d-7c9b-489b-92e9-6d0999f0c9ad\") " pod="e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt" Oct 30 13:48:39.905294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:39.905157 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-dshm\") pod \"simulator-single-main-f53955-kserve-86b865b85c-b99zt\" (UID: \"d4ce772d-7c9b-489b-92e9-6d0999f0c9ad\") " pod="e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt" Oct 30 13:48:39.905294 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:39.905195 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-tmp-dir\") pod \"simulator-single-main-f53955-kserve-86b865b85c-b99zt\" (UID: \"d4ce772d-7c9b-489b-92e9-6d0999f0c9ad\") " pod="e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt" Oct 30 13:48:39.905615 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:39.905537 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-home\") pod \"simulator-single-main-f53955-kserve-86b865b85c-b99zt\" (UID: \"d4ce772d-7c9b-489b-92e9-6d0999f0c9ad\") " pod="e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt" Oct 30 13:48:39.905683 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:39.905642 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-tmp-dir\") pod \"simulator-single-main-f53955-kserve-86b865b85c-b99zt\" (UID: \"d4ce772d-7c9b-489b-92e9-6d0999f0c9ad\") " pod="e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt" Oct 30 13:48:39.905789 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:39.905767 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-model-cache\") pod \"simulator-single-main-f53955-kserve-86b865b85c-b99zt\" (UID: \"d4ce772d-7c9b-489b-92e9-6d0999f0c9ad\") " pod="e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt" Oct 30 13:48:39.907459 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:39.907434 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-dshm\") pod \"simulator-single-main-f53955-kserve-86b865b85c-b99zt\" (UID: \"d4ce772d-7c9b-489b-92e9-6d0999f0c9ad\") " pod="e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt" Oct 30 13:48:39.907596 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:39.907510 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-tls-certs\") pod \"simulator-single-main-f53955-kserve-86b865b85c-b99zt\" (UID: \"d4ce772d-7c9b-489b-92e9-6d0999f0c9ad\") " pod="e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt" Oct 30 13:48:39.939768 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:39.939737 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljgqv\" (UniqueName: \"kubernetes.io/projected/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-kube-api-access-ljgqv\") pod \"simulator-single-main-f53955-kserve-86b865b85c-b99zt\" (UID: \"d4ce772d-7c9b-489b-92e9-6d0999f0c9ad\") " pod="e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt" Oct 30 13:48:40.013203 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:40.013102 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt" Oct 30 13:48:40.322275 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:40.322235 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt"] Oct 30 13:48:40.324667 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:48:40.324625 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4ce772d_7c9b_489b_92e9_6d0999f0c9ad.slice/crio-d8b3b04ad1aca1e16eba972e5315fed4cf66ac115d7b3c1b14ee9b7b326275b4 WatchSource:0}: Error finding container d8b3b04ad1aca1e16eba972e5315fed4cf66ac115d7b3c1b14ee9b7b326275b4: Status 404 returned error can't find the container with id d8b3b04ad1aca1e16eba972e5315fed4cf66ac115d7b3c1b14ee9b7b326275b4 Oct 30 13:48:41.233197 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:41.233148 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt" event={"ID":"d4ce772d-7c9b-489b-92e9-6d0999f0c9ad","Type":"ContainerStarted","Data":"c6ec7ef442f788a07da642bf86fc45e497edc21c75702e7ab997bb597a9c5d10"} Oct 30 13:48:41.233197 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:41.233202 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt" event={"ID":"d4ce772d-7c9b-489b-92e9-6d0999f0c9ad","Type":"ContainerStarted","Data":"d8b3b04ad1aca1e16eba972e5315fed4cf66ac115d7b3c1b14ee9b7b326275b4"} Oct 30 13:48:41.277371 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:41.277321 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt" podStartSLOduration=2.27730962 podStartE2EDuration="2.27730962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:41.276060955 +0000 UTC m=+3819.879559416" watchObservedRunningTime="2026-10-30 13:48:41.27730962 +0000 UTC m=+3819.880808066" Oct 30 13:48:42.013839 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:42.013810 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt" Oct 30 13:48:42.139249 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:42.139212 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:48:50.013943 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:50.013846 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt" Oct 30 13:48:50.026732 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:50.026703 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt" Oct 30 13:48:50.288098 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:50.288016 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt" Oct 30 13:48:57.826942 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:48:57.826900 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:49:03.885402 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:03.885363 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3901a94a-drbjn"] Oct 30 13:49:03.889135 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:03.889117 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3901a94a-drbjn" Oct 30 13:49:03.905546 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:03.905523 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:49:03.908613 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:03.908591 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:49:03.918599 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:03.918572 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:49:03.922493 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:03.922467 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3901a94a-drbjn"] Oct 30 13:49:03.932995 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:03.932966 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97c8f6f9-00c9-42a8-81b8-fe4f124350b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3901a94a-drbjn\" (UID: \"97c8f6f9-00c9-42a8-81b8-fe4f124350b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3901a94a-drbjn" Oct 30 13:49:03.933112 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:03.933007 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97c8f6f9-00c9-42a8-81b8-fe4f124350b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3901a94a-drbjn\" (UID: \"97c8f6f9-00c9-42a8-81b8-fe4f124350b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3901a94a-drbjn" Oct 30 13:49:03.933191 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:03.933162 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p5jl\" (UniqueName: \"kubernetes.io/projected/97c8f6f9-00c9-42a8-81b8-fe4f124350b5-kube-api-access-2p5jl\") pod \"maas-api-revoke-sub-3901a94a-drbjn\" (UID: \"97c8f6f9-00c9-42a8-81b8-fe4f124350b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3901a94a-drbjn" Oct 30 13:49:04.008771 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:04.008736 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:49:04.033992 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:04.033961 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97c8f6f9-00c9-42a8-81b8-fe4f124350b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3901a94a-drbjn\" (UID: \"97c8f6f9-00c9-42a8-81b8-fe4f124350b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3901a94a-drbjn" Oct 30 13:49:04.033992 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:04.033993 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97c8f6f9-00c9-42a8-81b8-fe4f124350b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3901a94a-drbjn\" (UID: \"97c8f6f9-00c9-42a8-81b8-fe4f124350b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3901a94a-drbjn" Oct 30 13:49:04.034207 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:04.034029 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p5jl\" (UniqueName: \"kubernetes.io/projected/97c8f6f9-00c9-42a8-81b8-fe4f124350b5-kube-api-access-2p5jl\") pod \"maas-api-revoke-sub-3901a94a-drbjn\" (UID: \"97c8f6f9-00c9-42a8-81b8-fe4f124350b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3901a94a-drbjn" Oct 30 13:49:04.034763 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:04.034741 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97c8f6f9-00c9-42a8-81b8-fe4f124350b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3901a94a-drbjn\" (UID: \"97c8f6f9-00c9-42a8-81b8-fe4f124350b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3901a94a-drbjn" Oct 30 13:49:04.036485 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:04.036466 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97c8f6f9-00c9-42a8-81b8-fe4f124350b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3901a94a-drbjn\" (UID: \"97c8f6f9-00c9-42a8-81b8-fe4f124350b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3901a94a-drbjn" Oct 30 13:49:04.058970 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:04.058947 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p5jl\" (UniqueName: \"kubernetes.io/projected/97c8f6f9-00c9-42a8-81b8-fe4f124350b5-kube-api-access-2p5jl\") pod \"maas-api-revoke-sub-3901a94a-drbjn\" (UID: \"97c8f6f9-00c9-42a8-81b8-fe4f124350b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3901a94a-drbjn" Oct 30 13:49:04.202296 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:04.202250 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3901a94a-drbjn" Oct 30 13:49:04.503059 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:04.503031 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3901a94a-drbjn"] Oct 30 13:49:04.504934 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:49:04.504906 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c8f6f9_00c9_42a8_81b8_fe4f124350b5.slice/crio-c1cdf33e14ff4bd802e235a54bf5d10b500c8313f453e326da2e1377cfb19ef9 WatchSource:0}: Error finding container c1cdf33e14ff4bd802e235a54bf5d10b500c8313f453e326da2e1377cfb19ef9: Status 404 returned error can't find the container with id c1cdf33e14ff4bd802e235a54bf5d10b500c8313f453e326da2e1377cfb19ef9 Oct 30 13:49:05.366073 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:05.366038 4931 generic.go:356] "Generic (PLEG): container finished" podID="97c8f6f9-00c9-42a8-81b8-fe4f124350b5" containerID="43792d6fb9ab08a7d741184c2ec1e7efa80e05be179f30037ade01fc92c264f9" exitCode=0 Oct 30 13:49:05.366530 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:05.366133 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3901a94a-drbjn" event={"ID":"97c8f6f9-00c9-42a8-81b8-fe4f124350b5","Type":"ContainerDied","Data":"43792d6fb9ab08a7d741184c2ec1e7efa80e05be179f30037ade01fc92c264f9"} Oct 30 13:49:05.366530 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:05.366235 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3901a94a-drbjn" event={"ID":"97c8f6f9-00c9-42a8-81b8-fe4f124350b5","Type":"ContainerStarted","Data":"c1cdf33e14ff4bd802e235a54bf5d10b500c8313f453e326da2e1377cfb19ef9"} Oct 30 13:49:06.503741 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:06.503711 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3901a94a-drbjn" Oct 30 13:49:06.552794 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:06.552766 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97c8f6f9-00c9-42a8-81b8-fe4f124350b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97c8f6f9-00c9-42a8-81b8-fe4f124350b5-maas-api-xks-ca\") pod \"97c8f6f9-00c9-42a8-81b8-fe4f124350b5\" (UID: \"97c8f6f9-00c9-42a8-81b8-fe4f124350b5\") " Oct 30 13:49:06.552932 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:06.552876 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97c8f6f9-00c9-42a8-81b8-fe4f124350b5-kube-api-access-2p5jl\" (UniqueName: \"kubernetes.io/projected/97c8f6f9-00c9-42a8-81b8-fe4f124350b5-kube-api-access-2p5jl\") pod \"97c8f6f9-00c9-42a8-81b8-fe4f124350b5\" (UID: \"97c8f6f9-00c9-42a8-81b8-fe4f124350b5\") " Oct 30 13:49:06.552932 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:06.552902 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97c8f6f9-00c9-42a8-81b8-fe4f124350b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97c8f6f9-00c9-42a8-81b8-fe4f124350b5-maas-api-openshift-ca\") pod \"97c8f6f9-00c9-42a8-81b8-fe4f124350b5\" (UID: \"97c8f6f9-00c9-42a8-81b8-fe4f124350b5\") " Oct 30 13:49:06.553439 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:06.553404 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97c8f6f9-00c9-42a8-81b8-fe4f124350b5-maas-api-openshift-ca" pod "97c8f6f9-00c9-42a8-81b8-fe4f124350b5" (UID: "97c8f6f9-00c9-42a8-81b8-fe4f124350b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:49:06.555122 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:06.555094 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97c8f6f9-00c9-42a8-81b8-fe4f124350b5-maas-api-xks-ca" pod "97c8f6f9-00c9-42a8-81b8-fe4f124350b5" (UID: "97c8f6f9-00c9-42a8-81b8-fe4f124350b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:49:06.555232 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:06.555187 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97c8f6f9-00c9-42a8-81b8-fe4f124350b5-kube-api-access-2p5jl" pod "97c8f6f9-00c9-42a8-81b8-fe4f124350b5" (UID: "97c8f6f9-00c9-42a8-81b8-fe4f124350b5"). InnerVolumeSpecName "kube-api-access-2p5jl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:49:06.654605 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:06.654537 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p5jl\" (UniqueName: \"kubernetes.io/projected/97c8f6f9-00c9-42a8-81b8-fe4f124350b5-kube-api-access-2p5jl\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:49:06.654605 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:06.654565 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97c8f6f9-00c9-42a8-81b8-fe4f124350b5-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:49:06.654605 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:06.654580 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97c8f6f9-00c9-42a8-81b8-fe4f124350b5-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:49:07.351710 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:49:07.351671 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:49:07.376943 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:07.376909 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3901a94a-drbjn" event={"ID":"97c8f6f9-00c9-42a8-81b8-fe4f124350b5","Type":"ContainerDied","Data":"c1cdf33e14ff4bd802e235a54bf5d10b500c8313f453e326da2e1377cfb19ef9"} Oct 30 13:49:07.377118 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:07.376959 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1cdf33e14ff4bd802e235a54bf5d10b500c8313f453e326da2e1377cfb19ef9" Oct 30 13:49:07.377118 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:07.376973 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3901a94a-drbjn" Oct 30 13:49:15.124247 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:15.124201 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f53955-xw8jx"] Oct 30 13:49:15.124752 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:15.124652 4931 cpu_manager.go:414] "containerMap: removing container" podUID="97c8f6f9-00c9-42a8-81b8-fe4f124350b5" containerName="revoke-keys" Oct 30 13:49:15.124752 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:15.124664 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97c8f6f9-00c9-42a8-81b8-fe4f124350b5" containerName="revoke-keys" Oct 30 13:49:15.124752 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:15.124722 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="97c8f6f9-00c9-42a8-81b8-fe4f124350b5" containerName="revoke-keys" Oct 30 13:49:15.127998 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:15.127978 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f53955-xw8jx" Oct 30 13:49:15.145501 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:15.145477 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:49:15.145709 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:15.145694 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:49:15.146165 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:15.146148 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:49:15.149443 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:15.149424 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:49:15.161740 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:15.161717 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f53955-xw8jx"] Oct 30 13:49:15.231830 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:15.231798 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bede4eea-7a49-4099-8f72-cad5357d3f48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f53955-xw8jx\" (UID: \"bede4eea-7a49-4099-8f72-cad5357d3f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f53955-xw8jx" Oct 30 13:49:15.231967 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:15.231930 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9pnh\" (UniqueName: \"kubernetes.io/projected/bede4eea-7a49-4099-8f72-cad5357d3f48-kube-api-access-b9pnh\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f53955-xw8jx\" (UID: \"bede4eea-7a49-4099-8f72-cad5357d3f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f53955-xw8jx" Oct 30 13:49:15.231967 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:15.231957 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bede4eea-7a49-4099-8f72-cad5357d3f48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f53955-xw8jx\" (UID: \"bede4eea-7a49-4099-8f72-cad5357d3f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f53955-xw8jx" Oct 30 13:49:15.249096 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:15.249068 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:49:15.332754 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:15.332720 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9pnh\" (UniqueName: \"kubernetes.io/projected/bede4eea-7a49-4099-8f72-cad5357d3f48-kube-api-access-b9pnh\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f53955-xw8jx\" (UID: \"bede4eea-7a49-4099-8f72-cad5357d3f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f53955-xw8jx" Oct 30 13:49:15.332876 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:15.332757 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bede4eea-7a49-4099-8f72-cad5357d3f48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f53955-xw8jx\" (UID: \"bede4eea-7a49-4099-8f72-cad5357d3f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f53955-xw8jx" Oct 30 13:49:15.332876 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:15.332795 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bede4eea-7a49-4099-8f72-cad5357d3f48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f53955-xw8jx\" (UID: \"bede4eea-7a49-4099-8f72-cad5357d3f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f53955-xw8jx" Oct 30 13:49:15.333472 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:15.333453 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bede4eea-7a49-4099-8f72-cad5357d3f48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f53955-xw8jx\" (UID: \"bede4eea-7a49-4099-8f72-cad5357d3f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f53955-xw8jx" Oct 30 13:49:15.335311 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:15.335291 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bede4eea-7a49-4099-8f72-cad5357d3f48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f53955-xw8jx\" (UID: \"bede4eea-7a49-4099-8f72-cad5357d3f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f53955-xw8jx" Oct 30 13:49:15.357183 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:15.357155 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9pnh\" (UniqueName: \"kubernetes.io/projected/bede4eea-7a49-4099-8f72-cad5357d3f48-kube-api-access-b9pnh\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f53955-xw8jx\" (UID: \"bede4eea-7a49-4099-8f72-cad5357d3f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f53955-xw8jx" Oct 30 13:49:15.440645 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:15.440585 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f53955-xw8jx" Oct 30 13:49:15.758714 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:15.758525 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f53955-xw8jx"] Oct 30 13:49:15.761279 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:49:15.761242 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbede4eea_7a49_4099_8f72_cad5357d3f48.slice/crio-25de908c8d67587e3383af0b2de9355b963f6d46fea1b2ea96349ebf88fc7fd1 WatchSource:0}: Error finding container 25de908c8d67587e3383af0b2de9355b963f6d46fea1b2ea96349ebf88fc7fd1: Status 404 returned error can't find the container with id 25de908c8d67587e3383af0b2de9355b963f6d46fea1b2ea96349ebf88fc7fd1 Oct 30 13:49:16.424849 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:16.424763 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f53955-xw8jx" event={"ID":"bede4eea-7a49-4099-8f72-cad5357d3f48","Type":"ContainerStarted","Data":"13833efd5a96b9fd7d6da4ecade9dc3a3361845b157c475c0c5627cd91e1877c"} Oct 30 13:49:16.424849 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:16.424817 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f53955-xw8jx" event={"ID":"bede4eea-7a49-4099-8f72-cad5357d3f48","Type":"ContainerStarted","Data":"25de908c8d67587e3383af0b2de9355b963f6d46fea1b2ea96349ebf88fc7fd1"} Oct 30 13:49:16.471400 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:16.471345 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f53955-xw8jx" podStartSLOduration=1.471332269 podStartE2EDuration="1.471332269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:16.469637947 +0000 UTC m=+3855.073136392" watchObservedRunningTime="2026-10-30 13:49:16.471332269 +0000 UTC m=+3855.074830784" Oct 30 13:49:17.432379 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:17.432344 4931 generic.go:356] "Generic (PLEG): container finished" podID="bede4eea-7a49-4099-8f72-cad5357d3f48" containerID="13833efd5a96b9fd7d6da4ecade9dc3a3361845b157c475c0c5627cd91e1877c" exitCode=0 Oct 30 13:49:17.432818 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:17.432406 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f53955-xw8jx" event={"ID":"bede4eea-7a49-4099-8f72-cad5357d3f48","Type":"ContainerDied","Data":"13833efd5a96b9fd7d6da4ecade9dc3a3361845b157c475c0c5627cd91e1877c"} Oct 30 13:49:18.574574 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:18.574544 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f53955-xw8jx" Oct 30 13:49:18.661917 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:18.661876 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bede4eea-7a49-4099-8f72-cad5357d3f48-kube-api-access-b9pnh\" (UniqueName: \"kubernetes.io/projected/bede4eea-7a49-4099-8f72-cad5357d3f48-kube-api-access-b9pnh\") pod \"bede4eea-7a49-4099-8f72-cad5357d3f48\" (UID: \"bede4eea-7a49-4099-8f72-cad5357d3f48\") " Oct 30 13:49:18.662087 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:18.661943 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bede4eea-7a49-4099-8f72-cad5357d3f48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bede4eea-7a49-4099-8f72-cad5357d3f48-maas-api-xks-ca\") pod \"bede4eea-7a49-4099-8f72-cad5357d3f48\" (UID: \"bede4eea-7a49-4099-8f72-cad5357d3f48\") " Oct 30 13:49:18.662087 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:18.662028 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bede4eea-7a49-4099-8f72-cad5357d3f48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bede4eea-7a49-4099-8f72-cad5357d3f48-maas-api-openshift-ca\") pod \"bede4eea-7a49-4099-8f72-cad5357d3f48\" (UID: \"bede4eea-7a49-4099-8f72-cad5357d3f48\") " Oct 30 13:49:18.662417 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:18.662391 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bede4eea-7a49-4099-8f72-cad5357d3f48-maas-api-openshift-ca" pod "bede4eea-7a49-4099-8f72-cad5357d3f48" (UID: "bede4eea-7a49-4099-8f72-cad5357d3f48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:49:18.664134 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:18.664103 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bede4eea-7a49-4099-8f72-cad5357d3f48-maas-api-xks-ca" pod "bede4eea-7a49-4099-8f72-cad5357d3f48" (UID: "bede4eea-7a49-4099-8f72-cad5357d3f48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:49:18.664233 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:18.664192 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bede4eea-7a49-4099-8f72-cad5357d3f48-kube-api-access-b9pnh" pod "bede4eea-7a49-4099-8f72-cad5357d3f48" (UID: "bede4eea-7a49-4099-8f72-cad5357d3f48"). InnerVolumeSpecName "kube-api-access-b9pnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:49:18.763608 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:18.763570 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bede4eea-7a49-4099-8f72-cad5357d3f48-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:49:18.763608 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:18.763607 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9pnh\" (UniqueName: \"kubernetes.io/projected/bede4eea-7a49-4099-8f72-cad5357d3f48-kube-api-access-b9pnh\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:49:18.763608 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:18.763617 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bede4eea-7a49-4099-8f72-cad5357d3f48-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:49:19.444352 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:19.444307 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f53955-xw8jx" event={"ID":"bede4eea-7a49-4099-8f72-cad5357d3f48","Type":"ContainerDied","Data":"25de908c8d67587e3383af0b2de9355b963f6d46fea1b2ea96349ebf88fc7fd1"} Oct 30 13:49:19.444538 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:19.444372 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25de908c8d67587e3383af0b2de9355b963f6d46fea1b2ea96349ebf88fc7fd1" Oct 30 13:49:19.444538 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:19.444323 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f53955-xw8jx" Oct 30 13:49:19.876094 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:19.876000 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt"] Oct 30 13:49:27.631471 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:27.631435 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f53955-xw8jx"] Oct 30 13:49:27.646098 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:27.646059 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f53955-xw8jx"] Oct 30 13:49:29.303178 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:29.303138 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:49:29.511307 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:29.511248 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bede4eea-7a49-4099-8f72-cad5357d3f48" path="/var/lib/kubelet/pods/bede4eea-7a49-4099-8f72-cad5357d3f48/volumes" Oct 30 13:49:34.896462 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:34.896400 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt" podUID="d4ce772d-7c9b-489b-92e9-6d0999f0c9ad" containerName="main" containerID="cri-o://c6ec7ef442f788a07da642bf86fc45e497edc21c75702e7ab997bb597a9c5d10" gracePeriod=45 Oct 30 13:49:35.142783 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:35.142755 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt" Oct 30 13:49:35.225188 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:35.225155 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-tls-certs\" (UniqueName: \"kubernetes.io/secret/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-tls-certs\") pod \"d4ce772d-7c9b-489b-92e9-6d0999f0c9ad\" (UID: \"d4ce772d-7c9b-489b-92e9-6d0999f0c9ad\") " Oct 30 13:49:35.225450 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:35.225229 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-home\" (UniqueName: \"kubernetes.io/empty-dir/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-home\") pod \"d4ce772d-7c9b-489b-92e9-6d0999f0c9ad\" (UID: \"d4ce772d-7c9b-489b-92e9-6d0999f0c9ad\") " Oct 30 13:49:35.225450 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:35.225278 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-model-cache\") pod \"d4ce772d-7c9b-489b-92e9-6d0999f0c9ad\" (UID: \"d4ce772d-7c9b-489b-92e9-6d0999f0c9ad\") " Oct 30 13:49:35.225450 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:35.225327 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-tmp-dir\") pod \"d4ce772d-7c9b-489b-92e9-6d0999f0c9ad\" (UID: \"d4ce772d-7c9b-489b-92e9-6d0999f0c9ad\") " Oct 30 13:49:35.225450 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:35.225382 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-dshm\") pod \"d4ce772d-7c9b-489b-92e9-6d0999f0c9ad\" (UID: \"d4ce772d-7c9b-489b-92e9-6d0999f0c9ad\") " Oct 30 13:49:35.225450 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:35.225414 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-kube-api-access-ljgqv\" (UniqueName: \"kubernetes.io/projected/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-kube-api-access-ljgqv\") pod \"d4ce772d-7c9b-489b-92e9-6d0999f0c9ad\" (UID: \"d4ce772d-7c9b-489b-92e9-6d0999f0c9ad\") " Oct 30 13:49:35.225765 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:35.225605 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-tmp-dir" pod "d4ce772d-7c9b-489b-92e9-6d0999f0c9ad" (UID: "d4ce772d-7c9b-489b-92e9-6d0999f0c9ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:49:35.225765 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:35.225615 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-model-cache" pod "d4ce772d-7c9b-489b-92e9-6d0999f0c9ad" (UID: "d4ce772d-7c9b-489b-92e9-6d0999f0c9ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:49:35.225890 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:35.225798 4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-model-cache\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:49:35.225890 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:35.225822 4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-tmp-dir\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:49:35.225890 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:35.225831 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-home" pod "d4ce772d-7c9b-489b-92e9-6d0999f0c9ad" (UID: "d4ce772d-7c9b-489b-92e9-6d0999f0c9ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:49:35.227684 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:35.227639 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-dshm" pod "d4ce772d-7c9b-489b-92e9-6d0999f0c9ad" (UID: "d4ce772d-7c9b-489b-92e9-6d0999f0c9ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:49:35.228031 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:35.228004 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-kube-api-access-ljgqv" pod "d4ce772d-7c9b-489b-92e9-6d0999f0c9ad" (UID: "d4ce772d-7c9b-489b-92e9-6d0999f0c9ad"). InnerVolumeSpecName "kube-api-access-ljgqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:49:35.228125 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:35.228010 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-tls-certs" pod "d4ce772d-7c9b-489b-92e9-6d0999f0c9ad" (UID: "d4ce772d-7c9b-489b-92e9-6d0999f0c9ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:49:35.327111 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:35.327069 4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-home\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:49:35.327111 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:35.327102 4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-dshm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:49:35.327111 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:35.327115 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljgqv\" (UniqueName: \"kubernetes.io/projected/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-kube-api-access-ljgqv\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:49:35.327480 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:35.327130 4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad-tls-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:49:35.528503 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:35.528474 4931 generic.go:356] "Generic (PLEG): container finished" podID="d4ce772d-7c9b-489b-92e9-6d0999f0c9ad" containerID="c6ec7ef442f788a07da642bf86fc45e497edc21c75702e7ab997bb597a9c5d10" exitCode=0 Oct 30 13:49:35.528635 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:35.528509 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt" event={"ID":"d4ce772d-7c9b-489b-92e9-6d0999f0c9ad","Type":"ContainerDied","Data":"c6ec7ef442f788a07da642bf86fc45e497edc21c75702e7ab997bb597a9c5d10"} Oct 30 13:49:35.528635 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:35.528533 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt" event={"ID":"d4ce772d-7c9b-489b-92e9-6d0999f0c9ad","Type":"ContainerDied","Data":"d8b3b04ad1aca1e16eba972e5315fed4cf66ac115d7b3c1b14ee9b7b326275b4"} Oct 30 13:49:35.528635 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:35.528511 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt" Oct 30 13:49:35.528635 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:35.528547 4931 scope.go:122] "RemoveContainer" containerID="c6ec7ef442f788a07da642bf86fc45e497edc21c75702e7ab997bb597a9c5d10" Oct 30 13:49:35.537476 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:35.537456 4931 scope.go:122] "RemoveContainer" containerID="c6ec7ef442f788a07da642bf86fc45e497edc21c75702e7ab997bb597a9c5d10" Oct 30 13:49:35.537725 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:49:35.537704 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6ec7ef442f788a07da642bf86fc45e497edc21c75702e7ab997bb597a9c5d10\": container with ID starting with c6ec7ef442f788a07da642bf86fc45e497edc21c75702e7ab997bb597a9c5d10 not found: ID does not exist" containerID="c6ec7ef442f788a07da642bf86fc45e497edc21c75702e7ab997bb597a9c5d10" Oct 30 13:49:35.537796 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:35.537737 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6ec7ef442f788a07da642bf86fc45e497edc21c75702e7ab997bb597a9c5d10"} err="failed to get container status \"c6ec7ef442f788a07da642bf86fc45e497edc21c75702e7ab997bb597a9c5d10\": rpc error: code = NotFound desc = could not find container \"c6ec7ef442f788a07da642bf86fc45e497edc21c75702e7ab997bb597a9c5d10\": container with ID starting with c6ec7ef442f788a07da642bf86fc45e497edc21c75702e7ab997bb597a9c5d10 not found: ID does not exist" Oct 30 13:49:35.583211 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:35.583184 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt"] Oct 30 13:49:35.589723 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:35.589701 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-f53955/simulator-single-main-f53955-kserve-86b865b85c-b99zt"] Oct 30 13:49:37.351390 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:49:37.351352 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:49:37.511101 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:49:37.511059 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4ce772d-7c9b-489b-92e9-6d0999f0c9ad" path="/var/lib/kubelet/pods/d4ce772d-7c9b-489b-92e9-6d0999f0c9ad/volumes" Oct 30 13:50:02.759923 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:50:02.759892 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/2.log" Oct 30 13:50:02.765835 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:50:02.765813 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/2.log" Oct 30 13:50:06.049973 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:50:06.049933 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2tbzn"] Oct 30 13:50:06.063241 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:50:06.063207 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2tbzn"] Oct 30 13:50:07.352457 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:50:07.352406 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:50:07.511750 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:50:07.511716 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f364dcb-cdc5-4f66-b0a7-8c72033404c7" path="/var/lib/kubelet/pods/5f364dcb-cdc5-4f66-b0a7-8c72033404c7/volumes" Oct 30 13:50:17.038357 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:50:17.038250 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fwl5"] Oct 30 13:50:17.041158 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:50:17.041135 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fwl5"] Oct 30 13:50:17.512296 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:50:17.512251 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="558d636e-5349-4555-8a27-c6cd7a81021b" path="/var/lib/kubelet/pods/558d636e-5349-4555-8a27-c6cd7a81021b/volumes" Oct 30 13:50:37.351354 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:50:37.351313 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:51:05.148221 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:51:05.148185 4931 scope.go:122] "RemoveContainer" containerID="79af1d3339dbf5d5ccdc61cbea53a8a53168e82dd339d7a1bfc783945296b1ed" Oct 30 13:51:07.351509 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:51:07.351466 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:51:14.522286 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:51:14.522227 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:51:29.194879 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:51:29.194843 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:51:37.352435 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:51:37.352397 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:51:56.096870 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:51:56.096832 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:52:05.174911 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:05.174877 4931 scope.go:122] "RemoveContainer" containerID="c933e54f5a12d672c0fa19b465d2b5a414f70e2984ec1e43cd72130290269673" Oct 30 13:52:07.351965 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:52:07.351927 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:52:08.339029 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:08.338984 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l"] Oct 30 13:52:08.339764 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:08.339732 4931 cpu_manager.go:414] "containerMap: removing container" podUID="bede4eea-7a49-4099-8f72-cad5357d3f48" containerName="revoke-keys" Oct 30 13:52:08.339879 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:08.339767 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bede4eea-7a49-4099-8f72-cad5357d3f48" containerName="revoke-keys" Oct 30 13:52:08.339879 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:08.339814 4931 cpu_manager.go:414] "containerMap: removing container" podUID="d4ce772d-7c9b-489b-92e9-6d0999f0c9ad" containerName="main" Oct 30 13:52:08.339879 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:08.339823 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4ce772d-7c9b-489b-92e9-6d0999f0c9ad" containerName="main" Oct 30 13:52:08.339985 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:08.339961 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="bede4eea-7a49-4099-8f72-cad5357d3f48" containerName="revoke-keys" Oct 30 13:52:08.340026 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:08.339983 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4ce772d-7c9b-489b-92e9-6d0999f0c9ad" containerName="main" Oct 30 13:52:08.345056 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:08.345026 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l" Oct 30 13:52:08.378514 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:08.378487 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-bf43af\"/\"kube-root-ca.crt\"" Oct 30 13:52:08.378884 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:08.378604 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-bf43af\"/\"default-dockercfg-lzd2h\"" Oct 30 13:52:08.380617 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:08.380596 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l"] Oct 30 13:52:08.384336 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:08.384314 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-bf43af\"/\"simulator-single-main-bf43af-kserve-self-signed-certs\"" Oct 30 13:52:08.384460 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:08.384412 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-bf43af\"/\"openshift-service-ca.crt\"" Oct 30 13:52:08.421494 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:08.421468 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s5qg\" (UniqueName: \"kubernetes.io/projected/edc94182-d092-4d9c-bde7-ec2d6405dc56-kube-api-access-8s5qg\") pod \"simulator-single-main-bf43af-kserve-5959bdf57b-64c7l\" (UID: \"edc94182-d092-4d9c-bde7-ec2d6405dc56\") " pod="e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l" Oct 30 13:52:08.421620 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:08.421498 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edc94182-d092-4d9c-bde7-ec2d6405dc56-dshm\") pod \"simulator-single-main-bf43af-kserve-5959bdf57b-64c7l\" (UID: \"edc94182-d092-4d9c-bde7-ec2d6405dc56\") " pod="e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l" Oct 30 13:52:08.421620 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:08.421519 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edc94182-d092-4d9c-bde7-ec2d6405dc56-model-cache\") pod \"simulator-single-main-bf43af-kserve-5959bdf57b-64c7l\" (UID: \"edc94182-d092-4d9c-bde7-ec2d6405dc56\") " pod="e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l" Oct 30 13:52:08.421758 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:08.421677 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc94182-d092-4d9c-bde7-ec2d6405dc56-tmp-dir\") pod \"simulator-single-main-bf43af-kserve-5959bdf57b-64c7l\" (UID: \"edc94182-d092-4d9c-bde7-ec2d6405dc56\") " pod="e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l" Oct 30 13:52:08.421758 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:08.421721 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edc94182-d092-4d9c-bde7-ec2d6405dc56-home\") pod \"simulator-single-main-bf43af-kserve-5959bdf57b-64c7l\" (UID: \"edc94182-d092-4d9c-bde7-ec2d6405dc56\") " pod="e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l" Oct 30 13:52:08.421849 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:08.421789 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edc94182-d092-4d9c-bde7-ec2d6405dc56-tls-certs\") pod \"simulator-single-main-bf43af-kserve-5959bdf57b-64c7l\" (UID: \"edc94182-d092-4d9c-bde7-ec2d6405dc56\") " pod="e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l" Oct 30 13:52:08.523019 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:08.522983 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s5qg\" (UniqueName: \"kubernetes.io/projected/edc94182-d092-4d9c-bde7-ec2d6405dc56-kube-api-access-8s5qg\") pod \"simulator-single-main-bf43af-kserve-5959bdf57b-64c7l\" (UID: \"edc94182-d092-4d9c-bde7-ec2d6405dc56\") " pod="e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l" Oct 30 13:52:08.523019 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:08.523021 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edc94182-d092-4d9c-bde7-ec2d6405dc56-dshm\") pod \"simulator-single-main-bf43af-kserve-5959bdf57b-64c7l\" (UID: \"edc94182-d092-4d9c-bde7-ec2d6405dc56\") " pod="e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l" Oct 30 13:52:08.523295 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:08.523044 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edc94182-d092-4d9c-bde7-ec2d6405dc56-model-cache\") pod \"simulator-single-main-bf43af-kserve-5959bdf57b-64c7l\" (UID: \"edc94182-d092-4d9c-bde7-ec2d6405dc56\") " pod="e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l" Oct 30 13:52:08.523295 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:08.523101 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc94182-d092-4d9c-bde7-ec2d6405dc56-tmp-dir\") pod \"simulator-single-main-bf43af-kserve-5959bdf57b-64c7l\" (UID: \"edc94182-d092-4d9c-bde7-ec2d6405dc56\") " pod="e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l" Oct 30 13:52:08.523295 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:08.523131 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edc94182-d092-4d9c-bde7-ec2d6405dc56-home\") pod \"simulator-single-main-bf43af-kserve-5959bdf57b-64c7l\" (UID: \"edc94182-d092-4d9c-bde7-ec2d6405dc56\") " pod="e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l" Oct 30 13:52:08.523480 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:08.523257 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edc94182-d092-4d9c-bde7-ec2d6405dc56-tls-certs\") pod \"simulator-single-main-bf43af-kserve-5959bdf57b-64c7l\" (UID: \"edc94182-d092-4d9c-bde7-ec2d6405dc56\") " pod="e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l" Oct 30 13:52:08.523581 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:08.523560 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edc94182-d092-4d9c-bde7-ec2d6405dc56-home\") pod \"simulator-single-main-bf43af-kserve-5959bdf57b-64c7l\" (UID: \"edc94182-d092-4d9c-bde7-ec2d6405dc56\") " pod="e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l" Oct 30 13:52:08.523641 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:08.523592 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edc94182-d092-4d9c-bde7-ec2d6405dc56-model-cache\") pod \"simulator-single-main-bf43af-kserve-5959bdf57b-64c7l\" (UID: \"edc94182-d092-4d9c-bde7-ec2d6405dc56\") " pod="e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l" Oct 30 13:52:08.523692 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:08.523667 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc94182-d092-4d9c-bde7-ec2d6405dc56-tmp-dir\") pod \"simulator-single-main-bf43af-kserve-5959bdf57b-64c7l\" (UID: \"edc94182-d092-4d9c-bde7-ec2d6405dc56\") " pod="e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l" Oct 30 13:52:08.525840 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:08.525812 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edc94182-d092-4d9c-bde7-ec2d6405dc56-tls-certs\") pod \"simulator-single-main-bf43af-kserve-5959bdf57b-64c7l\" (UID: \"edc94182-d092-4d9c-bde7-ec2d6405dc56\") " pod="e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l" Oct 30 13:52:08.525971 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:08.525812 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edc94182-d092-4d9c-bde7-ec2d6405dc56-dshm\") pod \"simulator-single-main-bf43af-kserve-5959bdf57b-64c7l\" (UID: \"edc94182-d092-4d9c-bde7-ec2d6405dc56\") " pod="e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l" Oct 30 13:52:08.548024 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:08.548000 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s5qg\" (UniqueName: \"kubernetes.io/projected/edc94182-d092-4d9c-bde7-ec2d6405dc56-kube-api-access-8s5qg\") pod \"simulator-single-main-bf43af-kserve-5959bdf57b-64c7l\" (UID: \"edc94182-d092-4d9c-bde7-ec2d6405dc56\") " pod="e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l" Oct 30 13:52:08.660686 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:08.660595 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l" Oct 30 13:52:08.966587 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:08.966559 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l"] Oct 30 13:52:08.968748 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:52:08.968717 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podedc94182_d092_4d9c_bde7_ec2d6405dc56.slice/crio-f73632af6cfa830f50e8f41dc9e4199cb8424ed00beb8f5bd04f5d1c8aa99779 WatchSource:0}: Error finding container f73632af6cfa830f50e8f41dc9e4199cb8424ed00beb8f5bd04f5d1c8aa99779: Status 404 returned error can't find the container with id f73632af6cfa830f50e8f41dc9e4199cb8424ed00beb8f5bd04f5d1c8aa99779 Oct 30 13:52:09.312549 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:09.312466 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l" event={"ID":"edc94182-d092-4d9c-bde7-ec2d6405dc56","Type":"ContainerStarted","Data":"4ff30c7b5a910143ec405f868f3d55815c28d3e1d8e63981996c53c3fd0c9673"} Oct 30 13:52:09.312549 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:09.312513 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l" event={"ID":"edc94182-d092-4d9c-bde7-ec2d6405dc56","Type":"ContainerStarted","Data":"f73632af6cfa830f50e8f41dc9e4199cb8424ed00beb8f5bd04f5d1c8aa99779"} Oct 30 13:52:09.359845 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:09.359796 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l" podStartSLOduration=1.35978253 podStartE2EDuration="1.35978253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:52:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:52:09.358573812 +0000 UTC m=+4027.962072259" watchObservedRunningTime="2026-10-30 13:52:09.35978253 +0000 UTC m=+4027.963280997" Oct 30 13:52:09.661307 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:09.661201 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l" Oct 30 13:52:10.925709 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:10.925667 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:52:18.661615 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:18.661577 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l" Oct 30 13:52:18.674241 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:18.674215 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l" Oct 30 13:52:19.380683 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:19.380649 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l" Oct 30 13:52:26.085763 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:26.085722 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:52:28.239101 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:28.239059 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c671f30-94m24"] Oct 30 13:52:28.242822 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:28.242798 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c671f30-94m24" Oct 30 13:52:28.258809 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:28.258785 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:52:28.272114 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:28.272084 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:52:28.272261 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:28.272212 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:52:28.272261 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:28.272244 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:52:28.274518 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:28.274497 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c671f30-94m24"] Oct 30 13:52:28.417364 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:28.417328 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2da355bf-272d-4396-b3fe-bff2409ee201-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c671f30-94m24\" (UID: \"2da355bf-272d-4396-b3fe-bff2409ee201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c671f30-94m24" Oct 30 13:52:28.417544 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:28.417369 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ghck\" (UniqueName: \"kubernetes.io/projected/2da355bf-272d-4396-b3fe-bff2409ee201-kube-api-access-8ghck\") pod \"maas-api-revoke-sub-3c671f30-94m24\" (UID: \"2da355bf-272d-4396-b3fe-bff2409ee201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c671f30-94m24" Oct 30 13:52:28.417544 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:28.417472 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2da355bf-272d-4396-b3fe-bff2409ee201-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c671f30-94m24\" (UID: \"2da355bf-272d-4396-b3fe-bff2409ee201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c671f30-94m24" Oct 30 13:52:28.518290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:28.518173 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ghck\" (UniqueName: \"kubernetes.io/projected/2da355bf-272d-4396-b3fe-bff2409ee201-kube-api-access-8ghck\") pod \"maas-api-revoke-sub-3c671f30-94m24\" (UID: \"2da355bf-272d-4396-b3fe-bff2409ee201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c671f30-94m24" Oct 30 13:52:28.518290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:28.518245 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2da355bf-272d-4396-b3fe-bff2409ee201-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c671f30-94m24\" (UID: \"2da355bf-272d-4396-b3fe-bff2409ee201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c671f30-94m24" Oct 30 13:52:28.518549 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:28.518405 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2da355bf-272d-4396-b3fe-bff2409ee201-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c671f30-94m24\" (UID: \"2da355bf-272d-4396-b3fe-bff2409ee201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c671f30-94m24" Oct 30 13:52:28.518953 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:28.518927 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2da355bf-272d-4396-b3fe-bff2409ee201-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c671f30-94m24\" (UID: \"2da355bf-272d-4396-b3fe-bff2409ee201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c671f30-94m24" Oct 30 13:52:28.520709 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:28.520688 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2da355bf-272d-4396-b3fe-bff2409ee201-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c671f30-94m24\" (UID: \"2da355bf-272d-4396-b3fe-bff2409ee201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c671f30-94m24" Oct 30 13:52:28.542642 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:28.542613 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ghck\" (UniqueName: \"kubernetes.io/projected/2da355bf-272d-4396-b3fe-bff2409ee201-kube-api-access-8ghck\") pod \"maas-api-revoke-sub-3c671f30-94m24\" (UID: \"2da355bf-272d-4396-b3fe-bff2409ee201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c671f30-94m24" Oct 30 13:52:28.555386 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:28.555360 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c671f30-94m24" Oct 30 13:52:28.852676 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:28.852652 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c671f30-94m24"] Oct 30 13:52:28.854878 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:52:28.854842 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da355bf_272d_4396_b3fe_bff2409ee201.slice/crio-2d5677de0187658f147f294ab027ecd2702db6943280869ec573b98c8dce7a1a WatchSource:0}: Error finding container 2d5677de0187658f147f294ab027ecd2702db6943280869ec573b98c8dce7a1a: Status 404 returned error can't find the container with id 2d5677de0187658f147f294ab027ecd2702db6943280869ec573b98c8dce7a1a Oct 30 13:52:29.430118 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:29.430022 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c671f30-94m24" event={"ID":"2da355bf-272d-4396-b3fe-bff2409ee201","Type":"ContainerStarted","Data":"b0a439b614dd23dddf70a528f8180af05faad3a793ef4a02c5bc77f7b57171dd"} Oct 30 13:52:29.430118 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:29.430080 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c671f30-94m24" event={"ID":"2da355bf-272d-4396-b3fe-bff2409ee201","Type":"ContainerStarted","Data":"2d5677de0187658f147f294ab027ecd2702db6943280869ec573b98c8dce7a1a"} Oct 30 13:52:29.473772 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:29.473724 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c671f30-94m24" podStartSLOduration=1.473709422 podStartE2EDuration="1.473709422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:52:29.472840427 +0000 UTC m=+4048.076338864" watchObservedRunningTime="2026-10-30 13:52:29.473709422 +0000 UTC m=+4048.077207868" Oct 30 13:52:30.437618 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:30.437581 4931 generic.go:356] "Generic (PLEG): container finished" podID="2da355bf-272d-4396-b3fe-bff2409ee201" containerID="b0a439b614dd23dddf70a528f8180af05faad3a793ef4a02c5bc77f7b57171dd" exitCode=0 Oct 30 13:52:30.438055 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:30.437645 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c671f30-94m24" event={"ID":"2da355bf-272d-4396-b3fe-bff2409ee201","Type":"ContainerDied","Data":"b0a439b614dd23dddf70a528f8180af05faad3a793ef4a02c5bc77f7b57171dd"} Oct 30 13:52:31.578943 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:31.578919 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c671f30-94m24" Oct 30 13:52:31.749435 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:31.749357 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2da355bf-272d-4396-b3fe-bff2409ee201-kube-api-access-8ghck\" (UniqueName: \"kubernetes.io/projected/2da355bf-272d-4396-b3fe-bff2409ee201-kube-api-access-8ghck\") pod \"2da355bf-272d-4396-b3fe-bff2409ee201\" (UID: \"2da355bf-272d-4396-b3fe-bff2409ee201\") " Oct 30 13:52:31.749435 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:31.749404 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2da355bf-272d-4396-b3fe-bff2409ee201-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2da355bf-272d-4396-b3fe-bff2409ee201-maas-api-xks-ca\") pod \"2da355bf-272d-4396-b3fe-bff2409ee201\" (UID: \"2da355bf-272d-4396-b3fe-bff2409ee201\") " Oct 30 13:52:31.749630 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:31.749443 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2da355bf-272d-4396-b3fe-bff2409ee201-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2da355bf-272d-4396-b3fe-bff2409ee201-maas-api-openshift-ca\") pod \"2da355bf-272d-4396-b3fe-bff2409ee201\" (UID: \"2da355bf-272d-4396-b3fe-bff2409ee201\") " Oct 30 13:52:31.749888 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:31.749855 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2da355bf-272d-4396-b3fe-bff2409ee201-maas-api-openshift-ca" pod "2da355bf-272d-4396-b3fe-bff2409ee201" (UID: "2da355bf-272d-4396-b3fe-bff2409ee201"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:52:31.751500 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:31.751474 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2da355bf-272d-4396-b3fe-bff2409ee201-maas-api-xks-ca" pod "2da355bf-272d-4396-b3fe-bff2409ee201" (UID: "2da355bf-272d-4396-b3fe-bff2409ee201"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:52:31.751608 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:31.751478 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da355bf-272d-4396-b3fe-bff2409ee201-kube-api-access-8ghck" pod "2da355bf-272d-4396-b3fe-bff2409ee201" (UID: "2da355bf-272d-4396-b3fe-bff2409ee201"). InnerVolumeSpecName "kube-api-access-8ghck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:52:31.851011 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:31.850984 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2da355bf-272d-4396-b3fe-bff2409ee201-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:52:31.851011 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:31.851009 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ghck\" (UniqueName: \"kubernetes.io/projected/2da355bf-272d-4396-b3fe-bff2409ee201-kube-api-access-8ghck\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:52:31.851149 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:31.851020 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2da355bf-272d-4396-b3fe-bff2409ee201-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:52:32.448895 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:32.448865 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c671f30-94m24" Oct 30 13:52:32.449108 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:32.448864 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c671f30-94m24" event={"ID":"2da355bf-272d-4396-b3fe-bff2409ee201","Type":"ContainerDied","Data":"2d5677de0187658f147f294ab027ecd2702db6943280869ec573b98c8dce7a1a"} Oct 30 13:52:32.449108 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:32.448991 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d5677de0187658f147f294ab027ecd2702db6943280869ec573b98c8dce7a1a" Oct 30 13:52:37.351564 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:52:37.351528 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:52:39.267225 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:39.267191 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf43af-r42fz"] Oct 30 13:52:39.267640 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:39.267571 4931 cpu_manager.go:414] "containerMap: removing container" podUID="2da355bf-272d-4396-b3fe-bff2409ee201" containerName="revoke-keys" Oct 30 13:52:39.267640 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:39.267584 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2da355bf-272d-4396-b3fe-bff2409ee201" containerName="revoke-keys" Oct 30 13:52:39.267735 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:39.267684 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="2da355bf-272d-4396-b3fe-bff2409ee201" containerName="revoke-keys" Oct 30 13:52:39.270628 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:39.270607 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf43af-r42fz" Oct 30 13:52:39.286800 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:39.286772 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:52:39.300304 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:39.300279 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:52:39.300304 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:39.300299 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:52:39.300469 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:39.300349 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:52:39.302687 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:39.302663 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf43af-r42fz"] Oct 30 13:52:39.318693 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:39.318665 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd42e9e0-4466-49d9-bcc8-c347950c222f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-bf43af-r42fz\" (UID: \"fd42e9e0-4466-49d9-bcc8-c347950c222f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf43af-r42fz" Oct 30 13:52:39.318795 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:39.318700 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9n2s\" (UniqueName: \"kubernetes.io/projected/fd42e9e0-4466-49d9-bcc8-c347950c222f-kube-api-access-d9n2s\") pod \"maas-api-revoke-keys-e2e-worker-single-main-bf43af-r42fz\" (UID: \"fd42e9e0-4466-49d9-bcc8-c347950c222f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf43af-r42fz" Oct 30 13:52:39.318891 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:39.318866 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd42e9e0-4466-49d9-bcc8-c347950c222f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-bf43af-r42fz\" (UID: \"fd42e9e0-4466-49d9-bcc8-c347950c222f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf43af-r42fz" Oct 30 13:52:39.420093 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:39.420062 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd42e9e0-4466-49d9-bcc8-c347950c222f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-bf43af-r42fz\" (UID: \"fd42e9e0-4466-49d9-bcc8-c347950c222f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf43af-r42fz" Oct 30 13:52:39.420293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:39.420119 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd42e9e0-4466-49d9-bcc8-c347950c222f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-bf43af-r42fz\" (UID: \"fd42e9e0-4466-49d9-bcc8-c347950c222f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf43af-r42fz" Oct 30 13:52:39.420293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:39.420141 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9n2s\" (UniqueName: \"kubernetes.io/projected/fd42e9e0-4466-49d9-bcc8-c347950c222f-kube-api-access-d9n2s\") pod \"maas-api-revoke-keys-e2e-worker-single-main-bf43af-r42fz\" (UID: \"fd42e9e0-4466-49d9-bcc8-c347950c222f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf43af-r42fz" Oct 30 13:52:39.420823 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:39.420795 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd42e9e0-4466-49d9-bcc8-c347950c222f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-bf43af-r42fz\" (UID: \"fd42e9e0-4466-49d9-bcc8-c347950c222f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf43af-r42fz" Oct 30 13:52:39.422522 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:39.422503 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd42e9e0-4466-49d9-bcc8-c347950c222f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-bf43af-r42fz\" (UID: \"fd42e9e0-4466-49d9-bcc8-c347950c222f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf43af-r42fz" Oct 30 13:52:39.444760 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:39.444731 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9n2s\" (UniqueName: \"kubernetes.io/projected/fd42e9e0-4466-49d9-bcc8-c347950c222f-kube-api-access-d9n2s\") pod \"maas-api-revoke-keys-e2e-worker-single-main-bf43af-r42fz\" (UID: \"fd42e9e0-4466-49d9-bcc8-c347950c222f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf43af-r42fz" Oct 30 13:52:39.583218 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:39.583134 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf43af-r42fz" Oct 30 13:52:39.718168 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:39.718129 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:52:39.883687 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:39.883650 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf43af-r42fz"] Oct 30 13:52:39.891887 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:52:39.886102 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd42e9e0_4466_49d9_bcc8_c347950c222f.slice/crio-7b6e9a7063c0a57602f61eded956b70b94eb5228d9fa9b0ac32a485776d43222 WatchSource:0}: Error finding container 7b6e9a7063c0a57602f61eded956b70b94eb5228d9fa9b0ac32a485776d43222: Status 404 returned error can't find the container with id 7b6e9a7063c0a57602f61eded956b70b94eb5228d9fa9b0ac32a485776d43222 Oct 30 13:52:40.494084 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:40.494047 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf43af-r42fz" event={"ID":"fd42e9e0-4466-49d9-bcc8-c347950c222f","Type":"ContainerStarted","Data":"b921553134499951230fcde827df125b80d0e7f5927a8004e05e9a93551d88ee"} Oct 30 13:52:40.494521 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:40.494104 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf43af-r42fz" event={"ID":"fd42e9e0-4466-49d9-bcc8-c347950c222f","Type":"ContainerStarted","Data":"7b6e9a7063c0a57602f61eded956b70b94eb5228d9fa9b0ac32a485776d43222"} Oct 30 13:52:40.540741 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:40.540690 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf43af-r42fz" podStartSLOduration=1.5406764320000002 podStartE2EDuration="1.540676432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:52:40.53988351 +0000 UTC m=+4059.143381958" watchObservedRunningTime="2026-10-30 13:52:40.540676432 +0000 UTC m=+4059.144174877" Oct 30 13:52:41.500577 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:41.500541 4931 generic.go:356] "Generic (PLEG): container finished" podID="fd42e9e0-4466-49d9-bcc8-c347950c222f" containerID="b921553134499951230fcde827df125b80d0e7f5927a8004e05e9a93551d88ee" exitCode=0 Oct 30 13:52:41.500984 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:41.500629 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf43af-r42fz" event={"ID":"fd42e9e0-4466-49d9-bcc8-c347950c222f","Type":"ContainerDied","Data":"b921553134499951230fcde827df125b80d0e7f5927a8004e05e9a93551d88ee"} Oct 30 13:52:42.642398 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:42.642371 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf43af-r42fz" Oct 30 13:52:42.648344 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:42.648320 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd42e9e0-4466-49d9-bcc8-c347950c222f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd42e9e0-4466-49d9-bcc8-c347950c222f-maas-api-xks-ca\") pod \"fd42e9e0-4466-49d9-bcc8-c347950c222f\" (UID: \"fd42e9e0-4466-49d9-bcc8-c347950c222f\") " Oct 30 13:52:42.648416 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:42.648394 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd42e9e0-4466-49d9-bcc8-c347950c222f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd42e9e0-4466-49d9-bcc8-c347950c222f-maas-api-openshift-ca\") pod \"fd42e9e0-4466-49d9-bcc8-c347950c222f\" (UID: \"fd42e9e0-4466-49d9-bcc8-c347950c222f\") " Oct 30 13:52:42.648725 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:42.648703 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd42e9e0-4466-49d9-bcc8-c347950c222f-maas-api-openshift-ca" pod "fd42e9e0-4466-49d9-bcc8-c347950c222f" (UID: "fd42e9e0-4466-49d9-bcc8-c347950c222f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:52:42.650497 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:42.650477 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd42e9e0-4466-49d9-bcc8-c347950c222f-maas-api-xks-ca" pod "fd42e9e0-4466-49d9-bcc8-c347950c222f" (UID: "fd42e9e0-4466-49d9-bcc8-c347950c222f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:52:42.749135 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:42.749096 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd42e9e0-4466-49d9-bcc8-c347950c222f-kube-api-access-d9n2s\" (UniqueName: \"kubernetes.io/projected/fd42e9e0-4466-49d9-bcc8-c347950c222f-kube-api-access-d9n2s\") pod \"fd42e9e0-4466-49d9-bcc8-c347950c222f\" (UID: \"fd42e9e0-4466-49d9-bcc8-c347950c222f\") " Oct 30 13:52:42.749315 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:42.749301 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd42e9e0-4466-49d9-bcc8-c347950c222f-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:52:42.749360 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:42.749315 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd42e9e0-4466-49d9-bcc8-c347950c222f-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:52:42.751288 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:42.751245 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd42e9e0-4466-49d9-bcc8-c347950c222f-kube-api-access-d9n2s" pod "fd42e9e0-4466-49d9-bcc8-c347950c222f" (UID: "fd42e9e0-4466-49d9-bcc8-c347950c222f"). InnerVolumeSpecName "kube-api-access-d9n2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:52:42.850072 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:42.850040 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9n2s\" (UniqueName: \"kubernetes.io/projected/fd42e9e0-4466-49d9-bcc8-c347950c222f-kube-api-access-d9n2s\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:52:43.513633 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:43.513601 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf43af-r42fz" Oct 30 13:52:43.513633 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:43.513610 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf43af-r42fz" event={"ID":"fd42e9e0-4466-49d9-bcc8-c347950c222f","Type":"ContainerDied","Data":"7b6e9a7063c0a57602f61eded956b70b94eb5228d9fa9b0ac32a485776d43222"} Oct 30 13:52:43.513869 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:43.513671 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b6e9a7063c0a57602f61eded956b70b94eb5228d9fa9b0ac32a485776d43222" Oct 30 13:52:44.054984 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:44.054946 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l"] Oct 30 13:52:52.000889 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:52.000854 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf43af-r42fz"] Oct 30 13:52:52.016155 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:52.016119 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf43af-r42fz"] Oct 30 13:52:53.511467 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:53.511435 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd42e9e0-4466-49d9-bcc8-c347950c222f" path="/var/lib/kubelet/pods/fd42e9e0-4466-49d9-bcc8-c347950c222f/volumes" Oct 30 13:52:53.635664 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:53.635622 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:52:59.075176 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:59.075120 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l" podUID="edc94182-d092-4d9c-bde7-ec2d6405dc56" containerName="main" containerID="cri-o://4ff30c7b5a910143ec405f868f3d55815c28d3e1d8e63981996c53c3fd0c9673" gracePeriod=45 Oct 30 13:52:59.330526 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:59.330470 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l" Oct 30 13:52:59.400198 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:59.400163 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edc94182-d092-4d9c-bde7-ec2d6405dc56-home\" (UniqueName: \"kubernetes.io/empty-dir/edc94182-d092-4d9c-bde7-ec2d6405dc56-home\") pod \"edc94182-d092-4d9c-bde7-ec2d6405dc56\" (UID: \"edc94182-d092-4d9c-bde7-ec2d6405dc56\") " Oct 30 13:52:59.400431 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:59.400206 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edc94182-d092-4d9c-bde7-ec2d6405dc56-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc94182-d092-4d9c-bde7-ec2d6405dc56-tmp-dir\") pod \"edc94182-d092-4d9c-bde7-ec2d6405dc56\" (UID: \"edc94182-d092-4d9c-bde7-ec2d6405dc56\") " Oct 30 13:52:59.400431 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:59.400259 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edc94182-d092-4d9c-bde7-ec2d6405dc56-kube-api-access-8s5qg\" (UniqueName: \"kubernetes.io/projected/edc94182-d092-4d9c-bde7-ec2d6405dc56-kube-api-access-8s5qg\") pod \"edc94182-d092-4d9c-bde7-ec2d6405dc56\" (UID: \"edc94182-d092-4d9c-bde7-ec2d6405dc56\") " Oct 30 13:52:59.400431 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:59.400313 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edc94182-d092-4d9c-bde7-ec2d6405dc56-dshm\" (UniqueName: \"kubernetes.io/empty-dir/edc94182-d092-4d9c-bde7-ec2d6405dc56-dshm\") pod \"edc94182-d092-4d9c-bde7-ec2d6405dc56\" (UID: \"edc94182-d092-4d9c-bde7-ec2d6405dc56\") " Oct 30 13:52:59.400431 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:59.400337 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edc94182-d092-4d9c-bde7-ec2d6405dc56-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edc94182-d092-4d9c-bde7-ec2d6405dc56-model-cache\") pod \"edc94182-d092-4d9c-bde7-ec2d6405dc56\" (UID: \"edc94182-d092-4d9c-bde7-ec2d6405dc56\") " Oct 30 13:52:59.400431 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:59.400379 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edc94182-d092-4d9c-bde7-ec2d6405dc56-tls-certs\" (UniqueName: \"kubernetes.io/secret/edc94182-d092-4d9c-bde7-ec2d6405dc56-tls-certs\") pod \"edc94182-d092-4d9c-bde7-ec2d6405dc56\" (UID: \"edc94182-d092-4d9c-bde7-ec2d6405dc56\") " Oct 30 13:52:59.400759 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:59.400494 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc94182-d092-4d9c-bde7-ec2d6405dc56-home" pod "edc94182-d092-4d9c-bde7-ec2d6405dc56" (UID: "edc94182-d092-4d9c-bde7-ec2d6405dc56"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:52:59.400759 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:59.400709 4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edc94182-d092-4d9c-bde7-ec2d6405dc56-home\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:52:59.400878 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:59.400765 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc94182-d092-4d9c-bde7-ec2d6405dc56-tmp-dir" pod "edc94182-d092-4d9c-bde7-ec2d6405dc56" (UID: "edc94182-d092-4d9c-bde7-ec2d6405dc56"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:52:59.400968 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:59.400944 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc94182-d092-4d9c-bde7-ec2d6405dc56-model-cache" pod "edc94182-d092-4d9c-bde7-ec2d6405dc56" (UID: "edc94182-d092-4d9c-bde7-ec2d6405dc56"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:52:59.402814 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:59.402786 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc94182-d092-4d9c-bde7-ec2d6405dc56-dshm" pod "edc94182-d092-4d9c-bde7-ec2d6405dc56" (UID: "edc94182-d092-4d9c-bde7-ec2d6405dc56"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:52:59.403173 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:59.403146 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edc94182-d092-4d9c-bde7-ec2d6405dc56-tls-certs" pod "edc94182-d092-4d9c-bde7-ec2d6405dc56" (UID: "edc94182-d092-4d9c-bde7-ec2d6405dc56"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:52:59.403338 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:59.403182 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edc94182-d092-4d9c-bde7-ec2d6405dc56-kube-api-access-8s5qg" pod "edc94182-d092-4d9c-bde7-ec2d6405dc56" (UID: "edc94182-d092-4d9c-bde7-ec2d6405dc56"). InnerVolumeSpecName "kube-api-access-8s5qg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:52:59.501507 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:59.501463 4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc94182-d092-4d9c-bde7-ec2d6405dc56-tmp-dir\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:52:59.501507 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:59.501507 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s5qg\" (UniqueName: \"kubernetes.io/projected/edc94182-d092-4d9c-bde7-ec2d6405dc56-kube-api-access-8s5qg\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:52:59.501735 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:59.501522 4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edc94182-d092-4d9c-bde7-ec2d6405dc56-dshm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:52:59.501735 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:59.501539 4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edc94182-d092-4d9c-bde7-ec2d6405dc56-model-cache\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:52:59.501735 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:59.501553 4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edc94182-d092-4d9c-bde7-ec2d6405dc56-tls-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:52:59.604391 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:59.604311 4931 generic.go:356] "Generic (PLEG): container finished" podID="edc94182-d092-4d9c-bde7-ec2d6405dc56" containerID="4ff30c7b5a910143ec405f868f3d55815c28d3e1d8e63981996c53c3fd0c9673" exitCode=0 Oct 30 13:52:59.604391 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:59.604353 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l" Oct 30 13:52:59.604603 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:59.604401 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l" event={"ID":"edc94182-d092-4d9c-bde7-ec2d6405dc56","Type":"ContainerDied","Data":"4ff30c7b5a910143ec405f868f3d55815c28d3e1d8e63981996c53c3fd0c9673"} Oct 30 13:52:59.604603 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:59.604450 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l" event={"ID":"edc94182-d092-4d9c-bde7-ec2d6405dc56","Type":"ContainerDied","Data":"f73632af6cfa830f50e8f41dc9e4199cb8424ed00beb8f5bd04f5d1c8aa99779"} Oct 30 13:52:59.604603 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:59.604470 4931 scope.go:122] "RemoveContainer" containerID="4ff30c7b5a910143ec405f868f3d55815c28d3e1d8e63981996c53c3fd0c9673" Oct 30 13:52:59.613455 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:59.613435 4931 scope.go:122] "RemoveContainer" containerID="4ff30c7b5a910143ec405f868f3d55815c28d3e1d8e63981996c53c3fd0c9673" Oct 30 13:52:59.613730 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:52:59.613709 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ff30c7b5a910143ec405f868f3d55815c28d3e1d8e63981996c53c3fd0c9673\": container with ID starting with 4ff30c7b5a910143ec405f868f3d55815c28d3e1d8e63981996c53c3fd0c9673 not found: ID does not exist" containerID="4ff30c7b5a910143ec405f868f3d55815c28d3e1d8e63981996c53c3fd0c9673" Oct 30 13:52:59.613785 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:59.613737 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ff30c7b5a910143ec405f868f3d55815c28d3e1d8e63981996c53c3fd0c9673"} err="failed to get container status \"4ff30c7b5a910143ec405f868f3d55815c28d3e1d8e63981996c53c3fd0c9673\": rpc error: code = NotFound desc = could not find container \"4ff30c7b5a910143ec405f868f3d55815c28d3e1d8e63981996c53c3fd0c9673\": container with ID starting with 4ff30c7b5a910143ec405f868f3d55815c28d3e1d8e63981996c53c3fd0c9673 not found: ID does not exist" Oct 30 13:52:59.659540 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:59.659518 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l"] Oct 30 13:52:59.665120 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:52:59.665099 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-bf43af/simulator-single-main-bf43af-kserve-5959bdf57b-64c7l"] Oct 30 13:53:01.511804 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:01.511770 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edc94182-d092-4d9c-bde7-ec2d6405dc56" path="/var/lib/kubelet/pods/edc94182-d092-4d9c-bde7-ec2d6405dc56/volumes" Oct 30 13:53:03.994847 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:03.994810 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r"] Oct 30 13:53:03.995320 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:03.995217 4931 cpu_manager.go:414] "containerMap: removing container" podUID="edc94182-d092-4d9c-bde7-ec2d6405dc56" containerName="main" Oct 30 13:53:03.995320 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:03.995231 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edc94182-d092-4d9c-bde7-ec2d6405dc56" containerName="main" Oct 30 13:53:03.995320 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:03.995242 4931 cpu_manager.go:414] "containerMap: removing container" podUID="fd42e9e0-4466-49d9-bcc8-c347950c222f" containerName="revoke-keys" Oct 30 13:53:03.995320 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:03.995247 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd42e9e0-4466-49d9-bcc8-c347950c222f" containerName="revoke-keys" Oct 30 13:53:03.995481 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:03.995347 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="edc94182-d092-4d9c-bde7-ec2d6405dc56" containerName="main" Oct 30 13:53:03.995481 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:03.995355 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd42e9e0-4466-49d9-bcc8-c347950c222f" containerName="revoke-keys" Oct 30 13:53:03.999777 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:03.999754 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r" Oct 30 13:53:04.018345 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:04.018298 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-58g6n\"" Oct 30 13:53:04.022438 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:04.022407 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r"] Oct 30 13:53:04.031903 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:04.031856 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 13:53:04.032045 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:04.031928 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-8959c618-kserve-self-signed-certs\"" Oct 30 13:53:04.032045 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:04.031957 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 13:53:04.037395 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:04.037371 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48ed0ef5-5b30-478e-a2fb-23f77250a962-home\") pod \"e2e-same-id-a-8959c618-kserve-769f948c49-mww5r\" (UID: \"48ed0ef5-5b30-478e-a2fb-23f77250a962\") " pod="llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r" Oct 30 13:53:04.037532 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:04.037429 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48ed0ef5-5b30-478e-a2fb-23f77250a962-tmp-dir\") pod \"e2e-same-id-a-8959c618-kserve-769f948c49-mww5r\" (UID: \"48ed0ef5-5b30-478e-a2fb-23f77250a962\") " pod="llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r" Oct 30 13:53:04.037584 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:04.037529 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48ed0ef5-5b30-478e-a2fb-23f77250a962-tls-certs\") pod \"e2e-same-id-a-8959c618-kserve-769f948c49-mww5r\" (UID: \"48ed0ef5-5b30-478e-a2fb-23f77250a962\") " pod="llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r" Oct 30 13:53:04.037649 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:04.037625 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vrk2\" (UniqueName: \"kubernetes.io/projected/48ed0ef5-5b30-478e-a2fb-23f77250a962-kube-api-access-2vrk2\") pod \"e2e-same-id-a-8959c618-kserve-769f948c49-mww5r\" (UID: \"48ed0ef5-5b30-478e-a2fb-23f77250a962\") " pod="llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r" Oct 30 13:53:04.037714 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:04.037696 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48ed0ef5-5b30-478e-a2fb-23f77250a962-model-cache\") pod \"e2e-same-id-a-8959c618-kserve-769f948c49-mww5r\" (UID: \"48ed0ef5-5b30-478e-a2fb-23f77250a962\") " pod="llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r" Oct 30 13:53:04.037787 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:04.037771 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48ed0ef5-5b30-478e-a2fb-23f77250a962-dshm\") pod \"e2e-same-id-a-8959c618-kserve-769f948c49-mww5r\" (UID: \"48ed0ef5-5b30-478e-a2fb-23f77250a962\") " pod="llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r" Oct 30 13:53:04.139117 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:04.139080 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48ed0ef5-5b30-478e-a2fb-23f77250a962-home\") pod \"e2e-same-id-a-8959c618-kserve-769f948c49-mww5r\" (UID: \"48ed0ef5-5b30-478e-a2fb-23f77250a962\") " pod="llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r" Oct 30 13:53:04.139304 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:04.139143 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48ed0ef5-5b30-478e-a2fb-23f77250a962-tmp-dir\") pod \"e2e-same-id-a-8959c618-kserve-769f948c49-mww5r\" (UID: \"48ed0ef5-5b30-478e-a2fb-23f77250a962\") " pod="llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r" Oct 30 13:53:04.139304 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:04.139181 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48ed0ef5-5b30-478e-a2fb-23f77250a962-tls-certs\") pod \"e2e-same-id-a-8959c618-kserve-769f948c49-mww5r\" (UID: \"48ed0ef5-5b30-478e-a2fb-23f77250a962\") " pod="llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r" Oct 30 13:53:04.139304 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:04.139230 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vrk2\" (UniqueName: \"kubernetes.io/projected/48ed0ef5-5b30-478e-a2fb-23f77250a962-kube-api-access-2vrk2\") pod \"e2e-same-id-a-8959c618-kserve-769f948c49-mww5r\" (UID: \"48ed0ef5-5b30-478e-a2fb-23f77250a962\") " pod="llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r" Oct 30 13:53:04.139304 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:04.139296 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48ed0ef5-5b30-478e-a2fb-23f77250a962-model-cache\") pod \"e2e-same-id-a-8959c618-kserve-769f948c49-mww5r\" (UID: \"48ed0ef5-5b30-478e-a2fb-23f77250a962\") " pod="llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r" Oct 30 13:53:04.139545 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:04.139342 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48ed0ef5-5b30-478e-a2fb-23f77250a962-dshm\") pod \"e2e-same-id-a-8959c618-kserve-769f948c49-mww5r\" (UID: \"48ed0ef5-5b30-478e-a2fb-23f77250a962\") " pod="llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r" Oct 30 13:53:04.139587 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:04.139565 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48ed0ef5-5b30-478e-a2fb-23f77250a962-home\") pod \"e2e-same-id-a-8959c618-kserve-769f948c49-mww5r\" (UID: \"48ed0ef5-5b30-478e-a2fb-23f77250a962\") " pod="llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r" Oct 30 13:53:04.139626 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:04.139581 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48ed0ef5-5b30-478e-a2fb-23f77250a962-tmp-dir\") pod \"e2e-same-id-a-8959c618-kserve-769f948c49-mww5r\" (UID: \"48ed0ef5-5b30-478e-a2fb-23f77250a962\") " pod="llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r" Oct 30 13:53:04.139824 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:04.139804 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48ed0ef5-5b30-478e-a2fb-23f77250a962-model-cache\") pod \"e2e-same-id-a-8959c618-kserve-769f948c49-mww5r\" (UID: \"48ed0ef5-5b30-478e-a2fb-23f77250a962\") " pod="llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r" Oct 30 13:53:04.141634 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:04.141605 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48ed0ef5-5b30-478e-a2fb-23f77250a962-dshm\") pod \"e2e-same-id-a-8959c618-kserve-769f948c49-mww5r\" (UID: \"48ed0ef5-5b30-478e-a2fb-23f77250a962\") " pod="llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r" Oct 30 13:53:04.142121 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:04.142100 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48ed0ef5-5b30-478e-a2fb-23f77250a962-tls-certs\") pod \"e2e-same-id-a-8959c618-kserve-769f948c49-mww5r\" (UID: \"48ed0ef5-5b30-478e-a2fb-23f77250a962\") " pod="llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r" Oct 30 13:53:04.164588 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:04.164562 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vrk2\" (UniqueName: \"kubernetes.io/projected/48ed0ef5-5b30-478e-a2fb-23f77250a962-kube-api-access-2vrk2\") pod \"e2e-same-id-a-8959c618-kserve-769f948c49-mww5r\" (UID: \"48ed0ef5-5b30-478e-a2fb-23f77250a962\") " pod="llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r" Oct 30 13:53:04.314610 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:04.314523 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r" Oct 30 13:53:04.617099 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:04.617009 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r"] Oct 30 13:53:04.619350 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:53:04.619323 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48ed0ef5_5b30_478e_a2fb_23f77250a962.slice/crio-87cf234d5753aea80cdc33cb34f56d4ffdca7212843551978e31a897bda240a2 WatchSource:0}: Error finding container 87cf234d5753aea80cdc33cb34f56d4ffdca7212843551978e31a897bda240a2: Status 404 returned error can't find the container with id 87cf234d5753aea80cdc33cb34f56d4ffdca7212843551978e31a897bda240a2 Oct 30 13:53:04.631904 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:04.631856 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r" event={"ID":"48ed0ef5-5b30-478e-a2fb-23f77250a962","Type":"ContainerStarted","Data":"87cf234d5753aea80cdc33cb34f56d4ffdca7212843551978e31a897bda240a2"} Oct 30 13:53:05.638545 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:05.638463 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r" event={"ID":"48ed0ef5-5b30-478e-a2fb-23f77250a962","Type":"ContainerStarted","Data":"8989c8ad4dbd9f44774b913bed75c1ec85be5f9efa7f65eab99fa1d06846d5d9"} Oct 30 13:53:05.682735 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:05.682682 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r" podStartSLOduration=2.682666761 podStartE2EDuration="2.682666761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:53:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:53:05.681821336 +0000 UTC m=+4084.285319782" watchObservedRunningTime="2026-10-30 13:53:05.682666761 +0000 UTC m=+4084.286165207" Oct 30 13:53:06.214155 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:06.214118 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:53:06.315406 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:06.315364 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r" Oct 30 13:53:07.359258 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:53:07.359209 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:53:14.315106 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:14.315019 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r" Oct 30 13:53:14.327722 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:14.327694 4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r" Oct 30 13:53:14.707240 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:14.707209 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r" Oct 30 13:53:24.488980 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:24.488936 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:53:37.350655 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:53:37.350611 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:53:38.890700 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:38.890658 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:53:49.014728 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:49.014687 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r"] Oct 30 13:53:50.824463 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:50.824421 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:53:57.231368 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:53:57.231331 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:54:04.039130 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:04.039085 4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r" podUID="48ed0ef5-5b30-478e-a2fb-23f77250a962" containerName="main" containerID="cri-o://8989c8ad4dbd9f44774b913bed75c1ec85be5f9efa7f65eab99fa1d06846d5d9" gracePeriod=45 Oct 30 13:54:04.288355 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:04.288328 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r" Oct 30 13:54:04.448132 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:04.448098 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48ed0ef5-5b30-478e-a2fb-23f77250a962-tls-certs\" (UniqueName: \"kubernetes.io/secret/48ed0ef5-5b30-478e-a2fb-23f77250a962-tls-certs\") pod \"48ed0ef5-5b30-478e-a2fb-23f77250a962\" (UID: \"48ed0ef5-5b30-478e-a2fb-23f77250a962\") " Oct 30 13:54:04.448346 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:04.448155 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48ed0ef5-5b30-478e-a2fb-23f77250a962-dshm\" (UniqueName: \"kubernetes.io/empty-dir/48ed0ef5-5b30-478e-a2fb-23f77250a962-dshm\") pod \"48ed0ef5-5b30-478e-a2fb-23f77250a962\" (UID: \"48ed0ef5-5b30-478e-a2fb-23f77250a962\") " Oct 30 13:54:04.448346 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:04.448182 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48ed0ef5-5b30-478e-a2fb-23f77250a962-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48ed0ef5-5b30-478e-a2fb-23f77250a962-model-cache\") pod \"48ed0ef5-5b30-478e-a2fb-23f77250a962\" (UID: \"48ed0ef5-5b30-478e-a2fb-23f77250a962\") " Oct 30 13:54:04.448346 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:04.448292 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48ed0ef5-5b30-478e-a2fb-23f77250a962-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48ed0ef5-5b30-478e-a2fb-23f77250a962-tmp-dir\") pod \"48ed0ef5-5b30-478e-a2fb-23f77250a962\" (UID: \"48ed0ef5-5b30-478e-a2fb-23f77250a962\") " Oct 30 13:54:04.448346 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:04.448328 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48ed0ef5-5b30-478e-a2fb-23f77250a962-kube-api-access-2vrk2\" (UniqueName: \"kubernetes.io/projected/48ed0ef5-5b30-478e-a2fb-23f77250a962-kube-api-access-2vrk2\") pod \"48ed0ef5-5b30-478e-a2fb-23f77250a962\" (UID: \"48ed0ef5-5b30-478e-a2fb-23f77250a962\") " Oct 30 13:54:04.448346 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:04.448346 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48ed0ef5-5b30-478e-a2fb-23f77250a962-home\" (UniqueName: \"kubernetes.io/empty-dir/48ed0ef5-5b30-478e-a2fb-23f77250a962-home\") pod \"48ed0ef5-5b30-478e-a2fb-23f77250a962\" (UID: \"48ed0ef5-5b30-478e-a2fb-23f77250a962\") " Oct 30 13:54:04.448658 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:04.448590 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48ed0ef5-5b30-478e-a2fb-23f77250a962-tmp-dir" pod "48ed0ef5-5b30-478e-a2fb-23f77250a962" (UID: "48ed0ef5-5b30-478e-a2fb-23f77250a962"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:54:04.448658 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:04.448584 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48ed0ef5-5b30-478e-a2fb-23f77250a962-model-cache" pod "48ed0ef5-5b30-478e-a2fb-23f77250a962" (UID: "48ed0ef5-5b30-478e-a2fb-23f77250a962"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:54:04.448775 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:04.448675 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48ed0ef5-5b30-478e-a2fb-23f77250a962-home" pod "48ed0ef5-5b30-478e-a2fb-23f77250a962" (UID: "48ed0ef5-5b30-478e-a2fb-23f77250a962"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:54:04.450646 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:04.450616 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48ed0ef5-5b30-478e-a2fb-23f77250a962-tls-certs" pod "48ed0ef5-5b30-478e-a2fb-23f77250a962" (UID: "48ed0ef5-5b30-478e-a2fb-23f77250a962"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:54:04.450753 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:04.450650 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48ed0ef5-5b30-478e-a2fb-23f77250a962-dshm" pod "48ed0ef5-5b30-478e-a2fb-23f77250a962" (UID: "48ed0ef5-5b30-478e-a2fb-23f77250a962"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:54:04.450753 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:04.450675 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48ed0ef5-5b30-478e-a2fb-23f77250a962-kube-api-access-2vrk2" pod "48ed0ef5-5b30-478e-a2fb-23f77250a962" (UID: "48ed0ef5-5b30-478e-a2fb-23f77250a962"). InnerVolumeSpecName "kube-api-access-2vrk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:54:04.549377 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:04.549347 4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48ed0ef5-5b30-478e-a2fb-23f77250a962-tmp-dir\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:54:04.549377 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:04.549374 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vrk2\" (UniqueName: \"kubernetes.io/projected/48ed0ef5-5b30-478e-a2fb-23f77250a962-kube-api-access-2vrk2\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:54:04.549538 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:04.549386 4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48ed0ef5-5b30-478e-a2fb-23f77250a962-home\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:54:04.549538 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:04.549399 4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48ed0ef5-5b30-478e-a2fb-23f77250a962-tls-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:54:04.549538 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:04.549410 4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48ed0ef5-5b30-478e-a2fb-23f77250a962-dshm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:54:04.549538 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:04.549421 4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48ed0ef5-5b30-478e-a2fb-23f77250a962-model-cache\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:54:05.052468 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:05.052436 4931 generic.go:356] "Generic (PLEG): container finished" podID="48ed0ef5-5b30-478e-a2fb-23f77250a962" containerID="8989c8ad4dbd9f44774b913bed75c1ec85be5f9efa7f65eab99fa1d06846d5d9" exitCode=0 Oct 30 13:54:05.052965 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:05.052482 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r" Oct 30 13:54:05.052965 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:05.052523 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r" event={"ID":"48ed0ef5-5b30-478e-a2fb-23f77250a962","Type":"ContainerDied","Data":"8989c8ad4dbd9f44774b913bed75c1ec85be5f9efa7f65eab99fa1d06846d5d9"} Oct 30 13:54:05.052965 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:05.052576 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r" event={"ID":"48ed0ef5-5b30-478e-a2fb-23f77250a962","Type":"ContainerDied","Data":"87cf234d5753aea80cdc33cb34f56d4ffdca7212843551978e31a897bda240a2"} Oct 30 13:54:05.052965 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:05.052603 4931 scope.go:122] "RemoveContainer" containerID="8989c8ad4dbd9f44774b913bed75c1ec85be5f9efa7f65eab99fa1d06846d5d9" Oct 30 13:54:05.062743 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:05.062724 4931 scope.go:122] "RemoveContainer" containerID="8989c8ad4dbd9f44774b913bed75c1ec85be5f9efa7f65eab99fa1d06846d5d9" Oct 30 13:54:05.062981 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:54:05.062960 4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8989c8ad4dbd9f44774b913bed75c1ec85be5f9efa7f65eab99fa1d06846d5d9\": container with ID starting with 8989c8ad4dbd9f44774b913bed75c1ec85be5f9efa7f65eab99fa1d06846d5d9 not found: ID does not exist" containerID="8989c8ad4dbd9f44774b913bed75c1ec85be5f9efa7f65eab99fa1d06846d5d9" Oct 30 13:54:05.063048 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:05.062993 4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8989c8ad4dbd9f44774b913bed75c1ec85be5f9efa7f65eab99fa1d06846d5d9"} err="failed to get container status \"8989c8ad4dbd9f44774b913bed75c1ec85be5f9efa7f65eab99fa1d06846d5d9\": rpc error: code = NotFound desc = could not find container \"8989c8ad4dbd9f44774b913bed75c1ec85be5f9efa7f65eab99fa1d06846d5d9\": container with ID starting with 8989c8ad4dbd9f44774b913bed75c1ec85be5f9efa7f65eab99fa1d06846d5d9 not found: ID does not exist" Oct 30 13:54:05.110376 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:05.110353 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r"] Oct 30 13:54:05.123572 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:05.123550 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-8959c618-kserve-769f948c49-mww5r"] Oct 30 13:54:05.511136 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:05.511102 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48ed0ef5-5b30-478e-a2fb-23f77250a962" path="/var/lib/kubelet/pods/48ed0ef5-5b30-478e-a2fb-23f77250a962/volumes" Oct 30 13:54:06.490601 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:06.490567 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:54:07.038850 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:07.038815 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3901a94a-drbjn"] Oct 30 13:54:07.052750 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:07.052714 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3901a94a-drbjn"] Oct 30 13:54:07.352599 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:54:07.352513 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:54:07.512157 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:07.512123 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97c8f6f9-00c9-42a8-81b8-fe4f124350b5" path="/var/lib/kubelet/pods/97c8f6f9-00c9-42a8-81b8-fe4f124350b5/volumes" Oct 30 13:54:10.710858 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:10.710821 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:54:29.214038 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:29.214002 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:54:37.353222 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:54:37.353176 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:54:39.296464 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:39.296424 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:54:47.821540 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:47.821457 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:54:54.497618 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:54.497576 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vmvvg"] Oct 30 13:54:54.498062 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:54.497982 4931 cpu_manager.go:414] "containerMap: removing container" podUID="48ed0ef5-5b30-478e-a2fb-23f77250a962" containerName="main" Oct 30 13:54:54.498062 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:54.497993 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48ed0ef5-5b30-478e-a2fb-23f77250a962" containerName="main" Oct 30 13:54:54.498062 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:54.498063 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="48ed0ef5-5b30-478e-a2fb-23f77250a962" containerName="main" Oct 30 13:54:54.501143 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:54.501125 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vmvvg" Oct 30 13:54:54.517521 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:54.517497 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:54:54.517727 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:54.517710 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:54:54.531226 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:54.531198 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:54:54.531379 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:54.531198 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:54:54.534875 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:54.534848 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vmvvg"] Oct 30 13:54:54.550081 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:54.550055 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8df1bf9f-ca5e-4960-9292-c761d6a2917b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-vmvvg\" (UID: \"8df1bf9f-ca5e-4960-9292-c761d6a2917b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vmvvg" Oct 30 13:54:54.550199 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:54.550111 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8df1bf9f-ca5e-4960-9292-c761d6a2917b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-vmvvg\" (UID: \"8df1bf9f-ca5e-4960-9292-c761d6a2917b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vmvvg" Oct 30 13:54:54.550317 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:54.550292 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scz6d\" (UniqueName: \"kubernetes.io/projected/8df1bf9f-ca5e-4960-9292-c761d6a2917b-kube-api-access-scz6d\") pod \"maas-api-revoke-sub-f03629db-vmvvg\" (UID: \"8df1bf9f-ca5e-4960-9292-c761d6a2917b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vmvvg" Oct 30 13:54:54.651776 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:54.651741 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scz6d\" (UniqueName: \"kubernetes.io/projected/8df1bf9f-ca5e-4960-9292-c761d6a2917b-kube-api-access-scz6d\") pod \"maas-api-revoke-sub-f03629db-vmvvg\" (UID: \"8df1bf9f-ca5e-4960-9292-c761d6a2917b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vmvvg" Oct 30 13:54:54.651987 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:54.651801 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8df1bf9f-ca5e-4960-9292-c761d6a2917b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-vmvvg\" (UID: \"8df1bf9f-ca5e-4960-9292-c761d6a2917b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vmvvg" Oct 30 13:54:54.651987 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:54.651868 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8df1bf9f-ca5e-4960-9292-c761d6a2917b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-vmvvg\" (UID: \"8df1bf9f-ca5e-4960-9292-c761d6a2917b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vmvvg" Oct 30 13:54:54.652603 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:54.652578 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8df1bf9f-ca5e-4960-9292-c761d6a2917b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-vmvvg\" (UID: \"8df1bf9f-ca5e-4960-9292-c761d6a2917b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vmvvg" Oct 30 13:54:54.654347 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:54.654320 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8df1bf9f-ca5e-4960-9292-c761d6a2917b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-vmvvg\" (UID: \"8df1bf9f-ca5e-4960-9292-c761d6a2917b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vmvvg" Oct 30 13:54:54.676971 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:54.676945 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scz6d\" (UniqueName: \"kubernetes.io/projected/8df1bf9f-ca5e-4960-9292-c761d6a2917b-kube-api-access-scz6d\") pod \"maas-api-revoke-sub-f03629db-vmvvg\" (UID: \"8df1bf9f-ca5e-4960-9292-c761d6a2917b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vmvvg" Oct 30 13:54:54.814318 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:54.814231 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vmvvg" Oct 30 13:54:55.116536 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:55.116507 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vmvvg"] Oct 30 13:54:55.118402 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:54:55.118367 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8df1bf9f_ca5e_4960_9292_c761d6a2917b.slice/crio-9d519f4ecfc311ed1f1063d6d8a495f43f5071bb387cc5e6afc9f6f154522138 WatchSource:0}: Error finding container 9d519f4ecfc311ed1f1063d6d8a495f43f5071bb387cc5e6afc9f6f154522138: Status 404 returned error can't find the container with id 9d519f4ecfc311ed1f1063d6d8a495f43f5071bb387cc5e6afc9f6f154522138 Oct 30 13:54:55.349599 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:55.349562 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vmvvg" event={"ID":"8df1bf9f-ca5e-4960-9292-c761d6a2917b","Type":"ContainerStarted","Data":"9d519f4ecfc311ed1f1063d6d8a495f43f5071bb387cc5e6afc9f6f154522138"} Oct 30 13:54:56.356048 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:56.355966 4931 generic.go:356] "Generic (PLEG): container finished" podID="8df1bf9f-ca5e-4960-9292-c761d6a2917b" containerID="0fa57d943097eb813c9651f83d8bd20a9399641230fae70696ac90e7f4da0371" exitCode=0 Oct 30 13:54:56.356451 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:56.356042 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vmvvg" event={"ID":"8df1bf9f-ca5e-4960-9292-c761d6a2917b","Type":"ContainerDied","Data":"0fa57d943097eb813c9651f83d8bd20a9399641230fae70696ac90e7f4da0371"} Oct 30 13:54:57.501885 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:57.501850 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vmvvg" Oct 30 13:54:57.580656 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:57.580620 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8df1bf9f-ca5e-4960-9292-c761d6a2917b-kube-api-access-scz6d\" (UniqueName: \"kubernetes.io/projected/8df1bf9f-ca5e-4960-9292-c761d6a2917b-kube-api-access-scz6d\") pod \"8df1bf9f-ca5e-4960-9292-c761d6a2917b\" (UID: \"8df1bf9f-ca5e-4960-9292-c761d6a2917b\") " Oct 30 13:54:57.580870 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:57.580708 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8df1bf9f-ca5e-4960-9292-c761d6a2917b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8df1bf9f-ca5e-4960-9292-c761d6a2917b-maas-api-openshift-ca\") pod \"8df1bf9f-ca5e-4960-9292-c761d6a2917b\" (UID: \"8df1bf9f-ca5e-4960-9292-c761d6a2917b\") " Oct 30 13:54:57.580870 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:57.580784 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8df1bf9f-ca5e-4960-9292-c761d6a2917b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8df1bf9f-ca5e-4960-9292-c761d6a2917b-maas-api-xks-ca\") pod \"8df1bf9f-ca5e-4960-9292-c761d6a2917b\" (UID: \"8df1bf9f-ca5e-4960-9292-c761d6a2917b\") " Oct 30 13:54:57.581218 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:57.581182 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8df1bf9f-ca5e-4960-9292-c761d6a2917b-maas-api-openshift-ca" pod "8df1bf9f-ca5e-4960-9292-c761d6a2917b" (UID: "8df1bf9f-ca5e-4960-9292-c761d6a2917b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:54:57.581376 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:57.581358 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8df1bf9f-ca5e-4960-9292-c761d6a2917b-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:54:57.583520 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:57.583490 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8df1bf9f-ca5e-4960-9292-c761d6a2917b-kube-api-access-scz6d" pod "8df1bf9f-ca5e-4960-9292-c761d6a2917b" (UID: "8df1bf9f-ca5e-4960-9292-c761d6a2917b"). InnerVolumeSpecName "kube-api-access-scz6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:54:57.583629 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:57.583560 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8df1bf9f-ca5e-4960-9292-c761d6a2917b-maas-api-xks-ca" pod "8df1bf9f-ca5e-4960-9292-c761d6a2917b" (UID: "8df1bf9f-ca5e-4960-9292-c761d6a2917b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:54:57.682723 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:57.682645 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scz6d\" (UniqueName: \"kubernetes.io/projected/8df1bf9f-ca5e-4960-9292-c761d6a2917b-kube-api-access-scz6d\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:54:57.682723 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:57.682674 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8df1bf9f-ca5e-4960-9292-c761d6a2917b-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:54:58.368433 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:58.368402 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vmvvg" Oct 30 13:54:58.368619 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:58.368404 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vmvvg" event={"ID":"8df1bf9f-ca5e-4960-9292-c761d6a2917b","Type":"ContainerDied","Data":"9d519f4ecfc311ed1f1063d6d8a495f43f5071bb387cc5e6afc9f6f154522138"} Oct 30 13:54:58.368619 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:54:58.368547 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d519f4ecfc311ed1f1063d6d8a495f43f5071bb387cc5e6afc9f6f154522138" Oct 30 13:55:00.787493 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:00.787453 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:55:02.796470 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:02.796440 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/2.log" Oct 30 13:55:02.808282 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:02.808246 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/2.log" Oct 30 13:55:05.226715 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:05.226680 4931 scope.go:122] "RemoveContainer" containerID="43792d6fb9ab08a7d741184c2ec1e7efa80e05be179f30037ade01fc92c264f9" Oct 30 13:55:06.215846 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:06.215810 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-47txv"] Oct 30 13:55:06.216216 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:06.216203 4931 cpu_manager.go:414] "containerMap: removing container" podUID="8df1bf9f-ca5e-4960-9292-c761d6a2917b" containerName="revoke-keys" Oct 30 13:55:06.216291 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:06.216217 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8df1bf9f-ca5e-4960-9292-c761d6a2917b" containerName="revoke-keys" Oct 30 13:55:06.216338 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:06.216309 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="8df1bf9f-ca5e-4960-9292-c761d6a2917b" containerName="revoke-keys" Oct 30 13:55:06.220610 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:06.220582 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-47txv" Oct 30 13:55:06.238632 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:06.238605 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:55:06.251901 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:06.251872 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:55:06.252077 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:06.251872 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:55:06.252153 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:06.251896 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:55:06.254192 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:06.254162 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-47txv"] Oct 30 13:55:06.297347 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:06.297312 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f89qf\" (UniqueName: \"kubernetes.io/projected/56e87668-8f4e-4932-bc1d-a1bb999ac398-kube-api-access-f89qf\") pod \"maas-api-revoke-sub-0165d7d2-47txv\" (UID: \"56e87668-8f4e-4932-bc1d-a1bb999ac398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-47txv" Oct 30 13:55:06.297519 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:06.297369 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56e87668-8f4e-4932-bc1d-a1bb999ac398-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-47txv\" (UID: \"56e87668-8f4e-4932-bc1d-a1bb999ac398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-47txv" Oct 30 13:55:06.297570 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:06.297524 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56e87668-8f4e-4932-bc1d-a1bb999ac398-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-47txv\" (UID: \"56e87668-8f4e-4932-bc1d-a1bb999ac398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-47txv" Oct 30 13:55:06.398904 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:06.398864 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56e87668-8f4e-4932-bc1d-a1bb999ac398-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-47txv\" (UID: \"56e87668-8f4e-4932-bc1d-a1bb999ac398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-47txv" Oct 30 13:55:06.399131 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:06.398932 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f89qf\" (UniqueName: \"kubernetes.io/projected/56e87668-8f4e-4932-bc1d-a1bb999ac398-kube-api-access-f89qf\") pod \"maas-api-revoke-sub-0165d7d2-47txv\" (UID: \"56e87668-8f4e-4932-bc1d-a1bb999ac398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-47txv" Oct 30 13:55:06.399131 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:06.398975 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56e87668-8f4e-4932-bc1d-a1bb999ac398-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-47txv\" (UID: \"56e87668-8f4e-4932-bc1d-a1bb999ac398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-47txv" Oct 30 13:55:06.399589 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:06.399565 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56e87668-8f4e-4932-bc1d-a1bb999ac398-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-47txv\" (UID: \"56e87668-8f4e-4932-bc1d-a1bb999ac398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-47txv" Oct 30 13:55:06.401329 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:06.401304 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56e87668-8f4e-4932-bc1d-a1bb999ac398-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-47txv\" (UID: \"56e87668-8f4e-4932-bc1d-a1bb999ac398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-47txv" Oct 30 13:55:06.424513 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:06.424480 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f89qf\" (UniqueName: \"kubernetes.io/projected/56e87668-8f4e-4932-bc1d-a1bb999ac398-kube-api-access-f89qf\") pod \"maas-api-revoke-sub-0165d7d2-47txv\" (UID: \"56e87668-8f4e-4932-bc1d-a1bb999ac398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-47txv" Oct 30 13:55:06.533319 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:06.533230 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-47txv" Oct 30 13:55:06.836503 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:06.836471 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-47txv"] Oct 30 13:55:06.839328 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:55:06.839300 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56e87668_8f4e_4932_bc1d_a1bb999ac398.slice/crio-fb4de30a47a24e152418b381f7af0039013edfb91d5fe72e37e359f9cb792051 WatchSource:0}: Error finding container fb4de30a47a24e152418b381f7af0039013edfb91d5fe72e37e359f9cb792051: Status 404 returned error can't find the container with id fb4de30a47a24e152418b381f7af0039013edfb91d5fe72e37e359f9cb792051 Oct 30 13:55:07.353403 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:55:07.353364 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:55:07.426836 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:07.426754 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-47txv" event={"ID":"56e87668-8f4e-4932-bc1d-a1bb999ac398","Type":"ContainerStarted","Data":"74052d3fa570a1f2848cf4045dd69fcd7f4af14b8ba569b4c4326e4dcb144b81"} Oct 30 13:55:07.426836 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:07.426809 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-47txv" event={"ID":"56e87668-8f4e-4932-bc1d-a1bb999ac398","Type":"ContainerStarted","Data":"fb4de30a47a24e152418b381f7af0039013edfb91d5fe72e37e359f9cb792051"} Oct 30 13:55:07.472279 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:07.472226 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-47txv" podStartSLOduration=1.472213018 podStartE2EDuration="1.472213018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:55:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:55:07.469195616 +0000 UTC m=+4206.072694062" watchObservedRunningTime="2026-10-30 13:55:07.472213018 +0000 UTC m=+4206.075711766" Oct 30 13:55:08.433745 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:08.433712 4931 generic.go:356] "Generic (PLEG): container finished" podID="56e87668-8f4e-4932-bc1d-a1bb999ac398" containerID="74052d3fa570a1f2848cf4045dd69fcd7f4af14b8ba569b4c4326e4dcb144b81" exitCode=0 Oct 30 13:55:08.434198 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:08.433755 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-47txv" event={"ID":"56e87668-8f4e-4932-bc1d-a1bb999ac398","Type":"ContainerDied","Data":"74052d3fa570a1f2848cf4045dd69fcd7f4af14b8ba569b4c4326e4dcb144b81"} Oct 30 13:55:09.581408 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:09.581376 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-47txv" Oct 30 13:55:09.728663 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:09.728622 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56e87668-8f4e-4932-bc1d-a1bb999ac398-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56e87668-8f4e-4932-bc1d-a1bb999ac398-maas-api-openshift-ca\") pod \"56e87668-8f4e-4932-bc1d-a1bb999ac398\" (UID: \"56e87668-8f4e-4932-bc1d-a1bb999ac398\") " Oct 30 13:55:09.728879 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:09.728736 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56e87668-8f4e-4932-bc1d-a1bb999ac398-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56e87668-8f4e-4932-bc1d-a1bb999ac398-maas-api-xks-ca\") pod \"56e87668-8f4e-4932-bc1d-a1bb999ac398\" (UID: \"56e87668-8f4e-4932-bc1d-a1bb999ac398\") " Oct 30 13:55:09.728970 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:09.728900 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56e87668-8f4e-4932-bc1d-a1bb999ac398-kube-api-access-f89qf\" (UniqueName: \"kubernetes.io/projected/56e87668-8f4e-4932-bc1d-a1bb999ac398-kube-api-access-f89qf\") pod \"56e87668-8f4e-4932-bc1d-a1bb999ac398\" (UID: \"56e87668-8f4e-4932-bc1d-a1bb999ac398\") " Oct 30 13:55:09.729120 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:09.729090 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56e87668-8f4e-4932-bc1d-a1bb999ac398-maas-api-openshift-ca" pod "56e87668-8f4e-4932-bc1d-a1bb999ac398" (UID: "56e87668-8f4e-4932-bc1d-a1bb999ac398"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:55:09.729345 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:09.729324 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56e87668-8f4e-4932-bc1d-a1bb999ac398-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:55:09.730945 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:09.730922 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56e87668-8f4e-4932-bc1d-a1bb999ac398-kube-api-access-f89qf" pod "56e87668-8f4e-4932-bc1d-a1bb999ac398" (UID: "56e87668-8f4e-4932-bc1d-a1bb999ac398"). InnerVolumeSpecName "kube-api-access-f89qf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:55:09.731018 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:09.730945 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56e87668-8f4e-4932-bc1d-a1bb999ac398-maas-api-xks-ca" pod "56e87668-8f4e-4932-bc1d-a1bb999ac398" (UID: "56e87668-8f4e-4932-bc1d-a1bb999ac398"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:55:09.830367 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:09.830330 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56e87668-8f4e-4932-bc1d-a1bb999ac398-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:55:09.830367 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:09.830362 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f89qf\" (UniqueName: \"kubernetes.io/projected/56e87668-8f4e-4932-bc1d-a1bb999ac398-kube-api-access-f89qf\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:55:10.446139 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:10.446106 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-47txv" Oct 30 13:55:10.446367 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:10.446139 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-47txv" event={"ID":"56e87668-8f4e-4932-bc1d-a1bb999ac398","Type":"ContainerDied","Data":"fb4de30a47a24e152418b381f7af0039013edfb91d5fe72e37e359f9cb792051"} Oct 30 13:55:10.446367 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:10.446187 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb4de30a47a24e152418b381f7af0039013edfb91d5fe72e37e359f9cb792051" Oct 30 13:55:17.211099 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:17.211061 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dl7bw"] Oct 30 13:55:17.211669 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:17.211646 4931 cpu_manager.go:414] "containerMap: removing container" podUID="56e87668-8f4e-4932-bc1d-a1bb999ac398" containerName="revoke-keys" Oct 30 13:55:17.211734 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:17.211673 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56e87668-8f4e-4932-bc1d-a1bb999ac398" containerName="revoke-keys" Oct 30 13:55:17.211848 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:17.211834 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="56e87668-8f4e-4932-bc1d-a1bb999ac398" containerName="revoke-keys" Oct 30 13:55:17.215625 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:17.215603 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dl7bw" Oct 30 13:55:17.247552 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:17.247526 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:55:17.247782 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:17.247747 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:55:17.247912 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:17.247870 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\"" Oct 30 13:55:17.247912 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:17.247870 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:55:17.250071 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:17.250048 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dl7bw"] Oct 30 13:55:17.297621 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:17.297588 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64999290-3b70-4d61-bac9-80a881f935ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-dl7bw\" (UID: \"64999290-3b70-4d61-bac9-80a881f935ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dl7bw" Oct 30 13:55:17.297795 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:17.297648 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vw97\" (UniqueName: \"kubernetes.io/projected/64999290-3b70-4d61-bac9-80a881f935ff-kube-api-access-9vw97\") pod \"maas-api-revoke-sub-6b19338f-dl7bw\" (UID: \"64999290-3b70-4d61-bac9-80a881f935ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dl7bw" Oct 30 13:55:17.297795 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:17.297750 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64999290-3b70-4d61-bac9-80a881f935ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-dl7bw\" (UID: \"64999290-3b70-4d61-bac9-80a881f935ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dl7bw" Oct 30 13:55:17.399104 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:17.399067 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vw97\" (UniqueName: \"kubernetes.io/projected/64999290-3b70-4d61-bac9-80a881f935ff-kube-api-access-9vw97\") pod \"maas-api-revoke-sub-6b19338f-dl7bw\" (UID: \"64999290-3b70-4d61-bac9-80a881f935ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dl7bw" Oct 30 13:55:17.399302 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:17.399156 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64999290-3b70-4d61-bac9-80a881f935ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-dl7bw\" (UID: \"64999290-3b70-4d61-bac9-80a881f935ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dl7bw" Oct 30 13:55:17.399302 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:17.399214 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64999290-3b70-4d61-bac9-80a881f935ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-dl7bw\" (UID: \"64999290-3b70-4d61-bac9-80a881f935ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dl7bw" Oct 30 13:55:17.399852 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:17.399820 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64999290-3b70-4d61-bac9-80a881f935ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-dl7bw\" (UID: \"64999290-3b70-4d61-bac9-80a881f935ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dl7bw" Oct 30 13:55:17.401610 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:17.401590 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64999290-3b70-4d61-bac9-80a881f935ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-dl7bw\" (UID: \"64999290-3b70-4d61-bac9-80a881f935ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dl7bw" Oct 30 13:55:17.424591 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:17.424559 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vw97\" (UniqueName: \"kubernetes.io/projected/64999290-3b70-4d61-bac9-80a881f935ff-kube-api-access-9vw97\") pod \"maas-api-revoke-sub-6b19338f-dl7bw\" (UID: \"64999290-3b70-4d61-bac9-80a881f935ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dl7bw" Oct 30 13:55:17.528714 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:17.528632 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dl7bw" Oct 30 13:55:17.828391 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:17.828357 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dl7bw"] Oct 30 13:55:17.830261 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:55:17.830231 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64999290_3b70_4d61_bac9_80a881f935ff.slice/crio-6c3a02ffd79aaf0868618ec993eb8fb927355a6fab1c994799aa113397855617 WatchSource:0}: Error finding container 6c3a02ffd79aaf0868618ec993eb8fb927355a6fab1c994799aa113397855617: Status 404 returned error can't find the container with id 6c3a02ffd79aaf0868618ec993eb8fb927355a6fab1c994799aa113397855617 Oct 30 13:55:18.488975 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:18.488940 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dl7bw" event={"ID":"64999290-3b70-4d61-bac9-80a881f935ff","Type":"ContainerStarted","Data":"a7ae614e58f7b1c401e0f6ac816b0ffc720111e81b2aff58b2bd87fdff048fb1"} Oct 30 13:55:18.489430 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:18.488999 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dl7bw" event={"ID":"64999290-3b70-4d61-bac9-80a881f935ff","Type":"ContainerStarted","Data":"6c3a02ffd79aaf0868618ec993eb8fb927355a6fab1c994799aa113397855617"} Oct 30 13:55:18.532776 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:18.532715 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dl7bw" podStartSLOduration=1.532698468 podStartE2EDuration="1.532698468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:55:18.532245548 +0000 UTC m=+4217.135744007" watchObservedRunningTime="2026-10-30 13:55:18.532698468 +0000 UTC m=+4217.136196914" Oct 30 13:55:19.495550 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:19.495512 4931 generic.go:356] "Generic (PLEG): container finished" podID="64999290-3b70-4d61-bac9-80a881f935ff" containerID="a7ae614e58f7b1c401e0f6ac816b0ffc720111e81b2aff58b2bd87fdff048fb1" exitCode=0 Oct 30 13:55:19.496041 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:19.495606 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dl7bw" event={"ID":"64999290-3b70-4d61-bac9-80a881f935ff","Type":"ContainerDied","Data":"a7ae614e58f7b1c401e0f6ac816b0ffc720111e81b2aff58b2bd87fdff048fb1"} Oct 30 13:55:20.653398 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:20.653369 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dl7bw" Oct 30 13:55:20.726911 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:20.726867 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64999290-3b70-4d61-bac9-80a881f935ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64999290-3b70-4d61-bac9-80a881f935ff-maas-api-xks-ca\") pod \"64999290-3b70-4d61-bac9-80a881f935ff\" (UID: \"64999290-3b70-4d61-bac9-80a881f935ff\") " Oct 30 13:55:20.727098 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:20.726969 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64999290-3b70-4d61-bac9-80a881f935ff-kube-api-access-9vw97\" (UniqueName: \"kubernetes.io/projected/64999290-3b70-4d61-bac9-80a881f935ff-kube-api-access-9vw97\") pod \"64999290-3b70-4d61-bac9-80a881f935ff\" (UID: \"64999290-3b70-4d61-bac9-80a881f935ff\") " Oct 30 13:55:20.727098 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:20.727065 4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64999290-3b70-4d61-bac9-80a881f935ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64999290-3b70-4d61-bac9-80a881f935ff-maas-api-openshift-ca\") pod \"64999290-3b70-4d61-bac9-80a881f935ff\" (UID: \"64999290-3b70-4d61-bac9-80a881f935ff\") " Oct 30 13:55:20.727446 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:20.727418 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64999290-3b70-4d61-bac9-80a881f935ff-maas-api-openshift-ca" pod "64999290-3b70-4d61-bac9-80a881f935ff" (UID: "64999290-3b70-4d61-bac9-80a881f935ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:55:20.729031 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:20.729007 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64999290-3b70-4d61-bac9-80a881f935ff-maas-api-xks-ca" pod "64999290-3b70-4d61-bac9-80a881f935ff" (UID: "64999290-3b70-4d61-bac9-80a881f935ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:55:20.729123 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:20.729060 4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64999290-3b70-4d61-bac9-80a881f935ff-kube-api-access-9vw97" pod "64999290-3b70-4d61-bac9-80a881f935ff" (UID: "64999290-3b70-4d61-bac9-80a881f935ff"). InnerVolumeSpecName "kube-api-access-9vw97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:55:20.828526 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:20.828450 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64999290-3b70-4d61-bac9-80a881f935ff-maas-api-openshift-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:55:20.828526 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:20.828479 4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64999290-3b70-4d61-bac9-80a881f935ff-maas-api-xks-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:55:20.828526 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:20.828489 4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vw97\" (UniqueName: \"kubernetes.io/projected/64999290-3b70-4d61-bac9-80a881f935ff-kube-api-access-9vw97\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Oct 30 13:55:21.509050 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:21.509009 4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dl7bw" Oct 30 13:55:21.511799 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:21.511767 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dl7bw" event={"ID":"64999290-3b70-4d61-bac9-80a881f935ff","Type":"ContainerDied","Data":"6c3a02ffd79aaf0868618ec993eb8fb927355a6fab1c994799aa113397855617"} Oct 30 13:55:21.511976 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:21.511811 4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c3a02ffd79aaf0868618ec993eb8fb927355a6fab1c994799aa113397855617" Oct 30 13:55:28.326057 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:28.326022 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hb5z9"] Oct 30 13:55:37.351527 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:55:37.351481 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:55:40.110472 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:40.110438 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-l879q_75424157-54c3-41fc-8e08-1f2a976e254c/manager/0.log" Oct 30 13:55:40.320011 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:40.319976 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-psvs6_5c0cc6a6-f4f0-41c9-a6a4-522c19211b21/manager/0.log" Oct 30 13:55:40.559278 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:40.559242 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-44c8c_f2677f24-72a2-4130-b530-4720923f33a8/manager/0.log" Oct 30 13:55:41.139569 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:41.139541 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-scw4n_888aade1-5070-4097-8127-43a53ce4a3c3/server/0.log" Oct 30 13:55:41.820866 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:41.820837 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c4494f88b-6wsdl_35aac1f4-f144-4455-97bb-6634f299bcd2/copy-manifests/0.log" Oct 30 13:55:41.851914 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:41.851887 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c4494f88b-6wsdl_35aac1f4-f144-4455-97bb-6634f299bcd2/manager/0.log" Oct 30 13:55:43.850391 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:43.850361 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-47txv_56e87668-8f4e-4932-bc1d-a1bb999ac398/revoke-keys/0.log" Oct 30 13:55:44.047301 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:44.047259 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3c671f30-94m24_2da355bf-272d-4396-b3fe-bff2409ee201/revoke-keys/0.log" Oct 30 13:55:44.639854 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:44.639818 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-dl7bw_64999290-3b70-4d61-bac9-80a881f935ff/revoke-keys/0.log" Oct 30 13:55:45.434827 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:45.434799 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-vmvvg_8df1bf9f-ca5e-4960-9292-c761d6a2917b/revoke-keys/0.log" Oct 30 13:55:47.987881 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:47.987842 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-9x5sm_f06f5bb4-b9de-4364-92b8-49a6c90125a3/kuadrant-console-plugin/0.log" Oct 30 13:55:48.455293 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:48.455249 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-hb5z9_b6fd15bb-fccd-4bf2-b86c-6b7dc183af33/limitador/0.log" Oct 30 13:55:49.486762 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:49.486727 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8_e4a08295-1052-4b1c-86ff-ca8729ed4dd8/istio-proxy/0.log" Oct 30 13:55:50.314584 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:50.314550 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-4vfv9_27bc72a1-afb4-4124-a6d6-95d6c1f0eb48/istio-proxy/0.log" Oct 30 13:55:51.142399 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:51.142361 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-ddspx_d126cf3a-50cd-45e4-b37f-b268fdbb3d13/istio-proxy/1.log" Oct 30 13:55:51.999622 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:51.999587 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-m4kcl_faa03672-ba31-4767-9617-b6e0f5809a1d/istio-proxy/0.log" Oct 30 13:55:52.819689 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:52.819658 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5db587c4bc-dttsw_1d06e291-b9d3-48f7-8fc2-f615440631d9/router/0.log" Oct 30 13:55:53.304526 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:55:53.304497 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-ddspx_d126cf3a-50cd-45e4-b37f-b268fdbb3d13/istio-proxy/0.log" Oct 30 13:56:05.311988 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:56:05.311951 4931 scope.go:122] "RemoveContainer" containerID="13833efd5a96b9fd7d6da4ecade9dc3a3361845b157c475c0c5627cd91e1877c" Oct 30 13:56:05.871513 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:56:05.871479 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-l879q_75424157-54c3-41fc-8e08-1f2a976e254c/manager/0.log" Oct 30 13:56:06.068056 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:56:06.068021 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-psvs6_5c0cc6a6-f4f0-41c9-a6a4-522c19211b21/manager/0.log" Oct 30 13:56:06.282598 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:56:06.282567 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-44c8c_f2677f24-72a2-4130-b530-4720923f33a8/manager/0.log" Oct 30 13:56:06.763749 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:56:06.763720 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-scw4n_888aade1-5070-4097-8127-43a53ce4a3c3/server/0.log" Oct 30 13:56:07.351221 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:56:07.351179 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:56:07.442667 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:56:07.442636 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c4494f88b-6wsdl_35aac1f4-f144-4455-97bb-6634f299bcd2/copy-manifests/0.log" Oct 30 13:56:07.473989 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:56:07.473960 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c4494f88b-6wsdl_35aac1f4-f144-4455-97bb-6634f299bcd2/manager/0.log" Oct 30 13:56:09.207822 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:56:09.207792 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-47txv_56e87668-8f4e-4932-bc1d-a1bb999ac398/revoke-keys/0.log" Oct 30 13:56:09.401777 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:56:09.401745 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3c671f30-94m24_2da355bf-272d-4396-b3fe-bff2409ee201/revoke-keys/0.log" Oct 30 13:56:09.995005 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:56:09.994973 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-dl7bw_64999290-3b70-4d61-bac9-80a881f935ff/revoke-keys/0.log" Oct 30 13:56:10.809360 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:56:10.809332 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-vmvvg_8df1bf9f-ca5e-4960-9292-c761d6a2917b/revoke-keys/0.log" Oct 30 13:56:12.907388 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:56:12.907310 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-9x5sm_f06f5bb4-b9de-4364-92b8-49a6c90125a3/kuadrant-console-plugin/0.log" Oct 30 13:56:13.403014 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:56:13.402982 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-hb5z9_b6fd15bb-fccd-4bf2-b86c-6b7dc183af33/limitador/0.log" Oct 30 13:56:14.218403 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:56:14.218371 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8_e4a08295-1052-4b1c-86ff-ca8729ed4dd8/istio-proxy/0.log" Oct 30 13:56:15.046042 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:56:15.046002 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-4vfv9_27bc72a1-afb4-4124-a6d6-95d6c1f0eb48/istio-proxy/0.log" Oct 30 13:56:15.897574 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:56:15.897540 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-ddspx_d126cf3a-50cd-45e4-b37f-b268fdbb3d13/istio-proxy/1.log" Oct 30 13:56:16.705346 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:56:16.705313 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-m4kcl_faa03672-ba31-4767-9617-b6e0f5809a1d/istio-proxy/0.log" Oct 30 13:56:17.523481 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:56:17.523451 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5db587c4bc-dttsw_1d06e291-b9d3-48f7-8fc2-f615440631d9/router/0.log" Oct 30 13:56:37.350846 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:56:37.350801 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:57:07.351707 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:57:07.351671 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:57:32.033387 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:32.033346 4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c671f30-94m24"] Oct 30 13:57:32.046561 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:32.046528 4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c671f30-94m24"] Oct 30 13:57:33.511123 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:33.511076 4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2da355bf-272d-4396-b3fe-bff2409ee201" path="/var/lib/kubelet/pods/2da355bf-272d-4396-b3fe-bff2409ee201/volumes" Oct 30 13:57:37.351247 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:57:37.351211 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:57:39.644060 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:39.644015 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cz4kh/must-gather-jz9gp"] Oct 30 13:57:39.644622 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:39.644601 4931 cpu_manager.go:414] "containerMap: removing container" podUID="64999290-3b70-4d61-bac9-80a881f935ff" containerName="revoke-keys" Oct 30 13:57:39.644695 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:39.644623 4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64999290-3b70-4d61-bac9-80a881f935ff" containerName="revoke-keys" Oct 30 13:57:39.644752 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:39.644738 4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="64999290-3b70-4d61-bac9-80a881f935ff" containerName="revoke-keys" Oct 30 13:57:39.648110 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:39.648087 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cz4kh/must-gather-jz9gp" Oct 30 13:57:39.665353 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:39.665330 4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cz4kh\"/\"default-dockercfg-2bzw5\"" Oct 30 13:57:39.665458 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:39.665364 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cz4kh\"/\"openshift-service-ca.crt\"" Oct 30 13:57:39.665458 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:39.665371 4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cz4kh\"/\"kube-root-ca.crt\"" Oct 30 13:57:39.686013 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:39.685987 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cz4kh/must-gather-jz9gp"] Oct 30 13:57:39.782139 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:39.782099 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c45sr\" (UniqueName: \"kubernetes.io/projected/901e15a7-0ec7-418e-bd7b-f9ac08f144a4-kube-api-access-c45sr\") pod \"must-gather-jz9gp\" (UID: \"901e15a7-0ec7-418e-bd7b-f9ac08f144a4\") " pod="openshift-must-gather-cz4kh/must-gather-jz9gp" Oct 30 13:57:39.782139 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:39.782141 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/901e15a7-0ec7-418e-bd7b-f9ac08f144a4-must-gather-output\") pod \"must-gather-jz9gp\" (UID: \"901e15a7-0ec7-418e-bd7b-f9ac08f144a4\") " pod="openshift-must-gather-cz4kh/must-gather-jz9gp" Oct 30 13:57:39.882760 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:39.882720 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c45sr\" (UniqueName: \"kubernetes.io/projected/901e15a7-0ec7-418e-bd7b-f9ac08f144a4-kube-api-access-c45sr\") pod \"must-gather-jz9gp\" (UID: \"901e15a7-0ec7-418e-bd7b-f9ac08f144a4\") " pod="openshift-must-gather-cz4kh/must-gather-jz9gp" Oct 30 13:57:39.882760 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:39.882763 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/901e15a7-0ec7-418e-bd7b-f9ac08f144a4-must-gather-output\") pod \"must-gather-jz9gp\" (UID: \"901e15a7-0ec7-418e-bd7b-f9ac08f144a4\") " pod="openshift-must-gather-cz4kh/must-gather-jz9gp" Oct 30 13:57:39.883168 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:39.883148 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/901e15a7-0ec7-418e-bd7b-f9ac08f144a4-must-gather-output\") pod \"must-gather-jz9gp\" (UID: \"901e15a7-0ec7-418e-bd7b-f9ac08f144a4\") " pod="openshift-must-gather-cz4kh/must-gather-jz9gp" Oct 30 13:57:39.907865 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:39.907799 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c45sr\" (UniqueName: \"kubernetes.io/projected/901e15a7-0ec7-418e-bd7b-f9ac08f144a4-kube-api-access-c45sr\") pod \"must-gather-jz9gp\" (UID: \"901e15a7-0ec7-418e-bd7b-f9ac08f144a4\") " pod="openshift-must-gather-cz4kh/must-gather-jz9gp" Oct 30 13:57:39.962260 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:39.962230 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cz4kh/must-gather-jz9gp" Oct 30 13:57:40.273555 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:40.273530 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cz4kh/must-gather-jz9gp"] Oct 30 13:57:40.275818 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:57:40.275787 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod901e15a7_0ec7_418e_bd7b_f9ac08f144a4.slice/crio-a121560e2416d2f4e3f95ff5b84e98eef5a075b64b1ee96a87e4bacf245ef13c WatchSource:0}: Error finding container a121560e2416d2f4e3f95ff5b84e98eef5a075b64b1ee96a87e4bacf245ef13c: Status 404 returned error can't find the container with id a121560e2416d2f4e3f95ff5b84e98eef5a075b64b1ee96a87e4bacf245ef13c Oct 30 13:57:40.277585 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:40.277567 4931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:57:41.269354 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:41.269226 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cz4kh/must-gather-jz9gp" event={"ID":"901e15a7-0ec7-418e-bd7b-f9ac08f144a4","Type":"ContainerStarted","Data":"50430a1eccadace8e0f1a51637aba3bd9a674f6d5f224fd1ac0d23b69c1600c3"} Oct 30 13:57:41.269354 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:41.269305 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cz4kh/must-gather-jz9gp" event={"ID":"901e15a7-0ec7-418e-bd7b-f9ac08f144a4","Type":"ContainerStarted","Data":"7f170b6e6eb73c496de7df39bc7bdef1f1a15d0055711e038117380fb19dbd8d"} Oct 30 13:57:41.269354 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:41.269323 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cz4kh/must-gather-jz9gp" event={"ID":"901e15a7-0ec7-418e-bd7b-f9ac08f144a4","Type":"ContainerStarted","Data":"a121560e2416d2f4e3f95ff5b84e98eef5a075b64b1ee96a87e4bacf245ef13c"} Oct 30 13:57:41.318401 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:41.318348 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-cz4kh/must-gather-jz9gp" podStartSLOduration=1.621438511 podStartE2EDuration="2.318332211s" totalImagesPullingTime="696.8937ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:57:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:57:41.316204386 +0000 UTC m=+4359.919702830" watchObservedRunningTime="2026-10-30 13:57:41.318332211 +0000 UTC m=+4359.921830661" Oct 30 13:57:44.096041 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:44.095963 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nk9fb_21ea018f-70e6-4ee3-952d-704089532abb/global-pull-secret-syncer/0.log" Oct 30 13:57:44.466335 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:44.466296 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-6xnwh_9c377915-c344-4791-baf6-4b286b9c019b/connection-checker/0.log" Oct 30 13:57:44.587765 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:44.587734 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-cv78r_ae9fe0c2-0d00-4891-8549-bc99f5373da9/connection-checker/0.log" Oct 30 13:57:44.729343 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:44.726370 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-rnflh_a3928331-ce67-4c37-9c5d-63b0da642a0f/connection-checker/0.log" Oct 30 13:57:45.216305 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:45.215126 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-k2fjm_37620bdd-11a4-4c12-8be1-9813a85d575b/konnectivity-agent/0.log" Oct 30 13:57:45.458287 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:45.457979 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-194.ec2.internal_7bc488b7d1d453086a471da9a46ef80c/haproxy/0.log" Oct 30 13:57:51.917941 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:51.917911 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-9x5sm_f06f5bb4-b9de-4364-92b8-49a6c90125a3/kuadrant-console-plugin/0.log" Oct 30 13:57:52.249897 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:52.249849 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-hb5z9_b6fd15bb-fccd-4bf2-b86c-6b7dc183af33/limitador/0.log" Oct 30 13:57:53.630289 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:53.629736 4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cz4kh/perf-node-gather-daemonset-ctb2v"] Oct 30 13:57:53.642290 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:53.639213 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-ctb2v" Oct 30 13:57:53.661287 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:53.657905 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cz4kh/perf-node-gather-daemonset-ctb2v"] Oct 30 13:57:53.775439 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:53.775407 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8fe2a476-3aa2-4c1b-b293-629b41733e52-podres\") pod \"perf-node-gather-daemonset-ctb2v\" (UID: \"8fe2a476-3aa2-4c1b-b293-629b41733e52\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-ctb2v" Oct 30 13:57:53.775439 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:53.775446 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fe2a476-3aa2-4c1b-b293-629b41733e52-sys\") pod \"perf-node-gather-daemonset-ctb2v\" (UID: \"8fe2a476-3aa2-4c1b-b293-629b41733e52\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-ctb2v" Oct 30 13:57:53.775693 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:53.775570 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8fe2a476-3aa2-4c1b-b293-629b41733e52-lib-modules\") pod \"perf-node-gather-daemonset-ctb2v\" (UID: \"8fe2a476-3aa2-4c1b-b293-629b41733e52\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-ctb2v" Oct 30 13:57:53.775693 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:53.775630 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8fe2a476-3aa2-4c1b-b293-629b41733e52-proc\") pod \"perf-node-gather-daemonset-ctb2v\" (UID: \"8fe2a476-3aa2-4c1b-b293-629b41733e52\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-ctb2v" Oct 30 13:57:53.775693 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:53.775672 4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6j2f\" (UniqueName: \"kubernetes.io/projected/8fe2a476-3aa2-4c1b-b293-629b41733e52-kube-api-access-c6j2f\") pod \"perf-node-gather-daemonset-ctb2v\" (UID: \"8fe2a476-3aa2-4c1b-b293-629b41733e52\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-ctb2v" Oct 30 13:57:53.876807 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:53.876765 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8fe2a476-3aa2-4c1b-b293-629b41733e52-lib-modules\") pod \"perf-node-gather-daemonset-ctb2v\" (UID: \"8fe2a476-3aa2-4c1b-b293-629b41733e52\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-ctb2v" Oct 30 13:57:53.877003 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:53.876825 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8fe2a476-3aa2-4c1b-b293-629b41733e52-proc\") pod \"perf-node-gather-daemonset-ctb2v\" (UID: \"8fe2a476-3aa2-4c1b-b293-629b41733e52\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-ctb2v" Oct 30 13:57:53.877003 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:53.876858 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6j2f\" (UniqueName: \"kubernetes.io/projected/8fe2a476-3aa2-4c1b-b293-629b41733e52-kube-api-access-c6j2f\") pod \"perf-node-gather-daemonset-ctb2v\" (UID: \"8fe2a476-3aa2-4c1b-b293-629b41733e52\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-ctb2v" Oct 30 13:57:53.877003 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:53.876896 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8fe2a476-3aa2-4c1b-b293-629b41733e52-podres\") pod \"perf-node-gather-daemonset-ctb2v\" (UID: \"8fe2a476-3aa2-4c1b-b293-629b41733e52\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-ctb2v" Oct 30 13:57:53.877003 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:53.876913 4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fe2a476-3aa2-4c1b-b293-629b41733e52-sys\") pod \"perf-node-gather-daemonset-ctb2v\" (UID: \"8fe2a476-3aa2-4c1b-b293-629b41733e52\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-ctb2v" Oct 30 13:57:53.877003 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:53.876926 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8fe2a476-3aa2-4c1b-b293-629b41733e52-lib-modules\") pod \"perf-node-gather-daemonset-ctb2v\" (UID: \"8fe2a476-3aa2-4c1b-b293-629b41733e52\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-ctb2v" Oct 30 13:57:53.877003 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:53.876994 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fe2a476-3aa2-4c1b-b293-629b41733e52-sys\") pod \"perf-node-gather-daemonset-ctb2v\" (UID: \"8fe2a476-3aa2-4c1b-b293-629b41733e52\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-ctb2v" Oct 30 13:57:53.877327 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:53.877019 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8fe2a476-3aa2-4c1b-b293-629b41733e52-podres\") pod \"perf-node-gather-daemonset-ctb2v\" (UID: \"8fe2a476-3aa2-4c1b-b293-629b41733e52\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-ctb2v" Oct 30 13:57:53.877327 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:53.877109 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8fe2a476-3aa2-4c1b-b293-629b41733e52-proc\") pod \"perf-node-gather-daemonset-ctb2v\" (UID: \"8fe2a476-3aa2-4c1b-b293-629b41733e52\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-ctb2v" Oct 30 13:57:53.902035 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:53.901967 4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6j2f\" (UniqueName: \"kubernetes.io/projected/8fe2a476-3aa2-4c1b-b293-629b41733e52-kube-api-access-c6j2f\") pod \"perf-node-gather-daemonset-ctb2v\" (UID: \"8fe2a476-3aa2-4c1b-b293-629b41733e52\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-ctb2v" Oct 30 13:57:53.971045 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:53.971011 4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-ctb2v" Oct 30 13:57:54.305478 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:54.305449 4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cz4kh/perf-node-gather-daemonset-ctb2v"] Oct 30 13:57:54.308745 ip-10-0-142-194 kubenswrapper[4931]: W1030 13:57:54.308703 4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8fe2a476_3aa2_4c1b_b293_629b41733e52.slice/crio-5ee0653cdec837e9262040e28f261fee792c2f0d101db94ff666ca1211e2b874 WatchSource:0}: Error finding container 5ee0653cdec837e9262040e28f261fee792c2f0d101db94ff666ca1211e2b874: Status 404 returned error can't find the container with id 5ee0653cdec837e9262040e28f261fee792c2f0d101db94ff666ca1211e2b874 Oct 30 13:57:54.429968 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:54.429928 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-ctb2v" event={"ID":"8fe2a476-3aa2-4c1b-b293-629b41733e52","Type":"ContainerStarted","Data":"5ee0653cdec837e9262040e28f261fee792c2f0d101db94ff666ca1211e2b874"} Oct 30 13:57:54.653021 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:54.652938 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-vjdmx_64bdbc16-e845-45da-8f34-f0d788a75eac/cluster-monitoring-operator/0.log" Oct 30 13:57:55.438527 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:55.438481 4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-ctb2v" event={"ID":"8fe2a476-3aa2-4c1b-b293-629b41733e52","Type":"ContainerStarted","Data":"e48b143bccdade8748b68b59a7dccaa97bd70b998236eb811db0a601d37d13c2"} Oct 30 13:57:55.438747 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:55.438591 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-ctb2v" Oct 30 13:57:55.482700 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:55.482631 4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-ctb2v" podStartSLOduration=2.482610897 podStartE2EDuration="2.482610897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:57:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:57:55.48169311 +0000 UTC m=+4374.085191556" watchObservedRunningTime="2026-10-30 13:57:55.482610897 +0000 UTC m=+4374.086109344" Oct 30 13:57:56.079644 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:56.079612 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qlkj4_91a082d9-7a7e-401e-9c9a-3615a462c3a1/node-exporter/0.log" Oct 30 13:57:56.196651 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:56.196617 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qlkj4_91a082d9-7a7e-401e-9c9a-3615a462c3a1/kube-rbac-proxy/0.log" Oct 30 13:57:56.313607 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:57:56.313578 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qlkj4_91a082d9-7a7e-401e-9c9a-3615a462c3a1/init-textfile/0.log" Oct 30 13:58:00.336212 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:58:00.336176 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-5bmsj_877cfa0b-b1c6-4455-b232-b1ca558b5084/networking-console-plugin/0.log" Oct 30 13:58:01.366243 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:58:01.366192 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/2.log" Oct 30 13:58:01.371070 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:58:01.371042 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v4mks_00ee6aae-7264-4c8d-a337-c7c9f1132e9c/console-operator/3.log" Oct 30 13:58:01.453986 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:58:01.453955 4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-ctb2v" Oct 30 13:58:03.582515 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:58:03.582487 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-c8pgj_93e1032c-06a8-4418-99b7-540d1a4f1232/volume-data-source-validator/0.log" Oct 30 13:58:06.037846 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:58:06.037811 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w52p8_7cb39685-b929-48bd-afd2-228862e62668/dns/0.log" Oct 30 13:58:06.146909 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:58:06.146866 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w52p8_7cb39685-b929-48bd-afd2-228862e62668/kube-rbac-proxy/0.log" Oct 30 13:58:06.505377 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:58:06.505334 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8_e4a08295-1052-4b1c-86ff-ca8729ed4dd8/istio-proxy/0.log" Oct 30 13:58:06.518203 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:58:06.518174 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gt8nx_24a00317-8fb9-41f3-8e1a-d1d985887725/dns-node-resolver/0.log" Oct 30 13:58:07.354698 ip-10-0-142-194 kubenswrapper[4931]: E1030 13:58:07.354592 4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b94k6" containerName="extractor" Oct 30 13:58:08.074685 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:58:08.074626 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rjhg6_49c39196-4e77-4bcf-86b1-705a263e394b/node-ca/0.log" Oct 30 13:58:08.538698 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:58:08.538671 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-4vfv9_27bc72a1-afb4-4124-a6d6-95d6c1f0eb48/istio-proxy/0.log" Oct 30 13:58:08.793986 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:58:08.793906 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-ddspx_d126cf3a-50cd-45e4-b37f-b268fdbb3d13/istio-proxy/0.log" Oct 30 13:58:08.795164 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:58:08.795145 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-ddspx_d126cf3a-50cd-45e4-b37f-b268fdbb3d13/istio-proxy/1.log" Oct 30 13:58:09.018159 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:58:09.018130 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-m4kcl_faa03672-ba31-4767-9617-b6e0f5809a1d/istio-proxy/0.log" Oct 30 13:58:09.272404 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:58:09.272375 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5db587c4bc-dttsw_1d06e291-b9d3-48f7-8fc2-f615440631d9/router/0.log" Oct 30 13:58:09.955011 ip-10-0-142-194 kubenswrapper[4931]: I1030 13:58:09.954977 4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8_e4a08295-1052-4b1c-86ff-ca8729ed4dd8/istio-proxy/0.log"